[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck
julia.lawall at lip6.fr
Mon Nov 13 17:44:04 CET 2017
On Tue, 14 Nov 2017, Masahiro Yamada wrote:
> Hi Julia,
> 2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall at lip6.fr>:
> > On Fri, 10 Nov 2017, Julia Lawall wrote:
> >> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> >> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> >> > "coccicheck failed" error messages.
> >> The question is where parallelism should be specified. Currently, make
> >> coccicheck picks up the number of cores on the machine and passes that to
> >> Coccinelle.
> >> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
> >> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
> >> while in practice that degrades the performance as compared to 40 or 80
> >> cores.
> >> On the other hand, if we use the make command line argument (-j), then we
> >> will only get parallelism up to the number of semantic patches. Since
> >> some finish quickly, there will be a lot of wasted cycles.
> >> The best would be that the user knows what works well for his machine, and
> >> specifies it on the command line, and then that value gets propagated to
> >> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
> >> parallel but instead would cause Coccinelle to run one semantic patch on 8
> >> files in parallel. But I don't know if that can be done.
> > Sorry for these fairly nonsensical comments. make -j is going to consider
> > every file, then parse and run every semantic patch on that file. If the
> > parallelism is pushed down into Coccinelle, each semantic patch will be
> > parsed only once, and then Coccinelle will choose the files for which it
> > is relevant. If indexing is used (idutils, glimpse), then for semantic
> > patches that focus on specific keywords, Coccinelle will efficiently
> > ignore files that are not relevant. I don't think there would be many
> > cases where make -j would win. Perhaps it would be possible to detect
> > its used and abort with an appopriate message?
> I am afraid you and I are talking different things.
> For a usual usage of coccicheck, only one thread runs scripts/coccicheck
> even if -j is passed from the command line.
> coccicheck provides "J" to specify parallelism.
> if [ -z "$J" ]; then
> NPROC=$(getconf _NPROCESSORS_ONLN)
Even if J is not specified, then it still runs with the maximum number of
By default, coccicheck tries to run as parallel as possible.
Indeed, J= does set the number of threads speficied.
> My patch addresses a problem where coccicheck is used as CHECK.
> The default of CHECK is "sparse", but you can use any checker tool.
> In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel
> under control of GNU Make, so scripts/coccicheck is also invoked from
> multiple threads.
> Passing --jobs to spatch is not sensible because it checks only one file.
OK. I tried a simple make coccicheck -j4 and indeed it does not seem to
be complaining. The number of spatch processes goes over 160 though.
More information about the Cocci