pipeline - Makefile with variable number of targets -


i attempting data pipeline makefile. have big file want split in smaller pieces process in parallel. number of subsets , size of each subset not known beforehand. example, file

$ in {1..100}; echo $i >> a.txt; done

the first step in makefile should compute ranges,... lets make them fixed 

ranges.txt: a.txt    or in 0 25 50 75; echo  $$(($$i+1))'\t'$$(($$i+25)) >> $@; done

next step should read ranges.txt, , create target file each range in ranges.txt, a_1.txt, a_2.txt, a_3.txt, a_4.txt. a_1.txt contains lines 1 through 25, a_2.txt lines 26-50, , on... can done?

you don't version of make you're using, i'll assume gnu make. there few ways of doing things this; wrote set of blog posts metaprogramming in gnu make (by mean having make generate own rules automatically).

if me i'd use constructed include files method this. so, have rule above ranges.txt instead create makefile, perhaps ranges.mk. makefile contain set of targets such a_1.txt, a_2.txt, etc. , define target-specific variables defining start , stop values. can -include generated ranges.mk , make rebuild it. 1 thing haven't described when want recompute ranges: depend on contents of a.txt?

anyway, like:

.phony: all:  ranges.mk: a.txt   # really? why?         in 0 25 50 75; \             echo 'a_$$i.txt : range_start := $$(($$i+1))'; \             echo 'a_$$i.txt : range_end   := $$(($$i+25))'; \             echo 'targets += a_$$i.txt'; \         done > $@  -include ranges.mk  all: $(targets) $(targets) : a.txt   # seems more         process --out $@ --in $< --start $(range_start) --end $(range_end)

(or whatever command; don't give example).


Comments

Post a Comment

Popular posts from this blog

cakephp - simple blog with croogo -

i not problem, made simple blog in croogo, problem is not adding or modifying or delete, because arabic language ?? function add in controller `public function admin_add() { $this->set('title_for_layout', __('add part')); if (!empty($this->request->data)) { $this->part->create(); if ($this->part->save($this->request->data)) { $this->session->setflash(__('the part has been saved'), 'default', array('class' => 'success')); $this->redirect(array('action' => 'index'));//, $this->part->id)); } else { $this->session->setflash(__('the part not saved. please, try again.'), 'default', array('class' => 'error')); } } }` => model `class part extends appmodel { public $name = 'part'; var $belongst...
Read more

How to group boxplot outliers in gnuplot -

i have large set of data points. try plot them boxplot, of outliers exact same value , represented on line beside each other. found how set horizontal distance between outliers in gnuplot boxplot , doesn't much, apparently not possible. is possible group outliers together, print 1 point , print number in brackets beside indicate how many points there are? think make more readable in graph. for information, have 3 boxplots 1 x value , times 6 in 1 graph. using gnuplot 5 , played around pointsize, doesn't reduce distance anymore. hope can help! edit: set terminal pdf set output 'dat.pdf' file0 = 'dat1.dat' file1 = 'dat2.dat' file2 = 'dat3.dat' set pointsize 0.2 set notitle set xlabel 'x' set ylabel 'y' header = system('head -1 '.file0); n = words(header) set xtics ('' 1) set [i=1:n] xtics add (word(header, i) i) set style data boxplot plot file0 using (1-0.25):1:(0.2) boxplot lw 2 lc rgb '#8b0000...
Read more

bash - Performing variable substitution in a string -

i have string contains use of variable, , i'd substitute variable's value string. right best have is #!/bin/bash foo='$name; echo bar' name="the name" expanded="$(eval echo "$foo")" echo "$expanded" which has obvious defects: prints the name bar while i'd print the name; echo bar instead of eval can bash's regex matching bash's string replacement: foo='$name; echo bar' name="the name" [[ "$foo" =~ \$([[:alnum:]]+) ]] && s="${!bash_rematch[1]}" && expanded="${foo/\$${bash_rematch[1]}/$s}" echo "$expanded" name; echo bar
Read more