poison mmap cache

[James K. Lowden]

I'm testing different algorithms for performance against a single
memory-mapped file. How can I best ensure that for each run the
page cache is cold, that the prior run left nothing for the current run
to take advantage of?

I tried calling mmap(2) on an unrelated file larger than RAM, and
iterating over it. Not only would I prefer something more
deterministic, but the following output suggests to me that technique
wasn't effective:

begin= 0x7f7f9de1e000, n= 10000000
sorted 1 of 100000000 in 2 seconds
begin= 0x7f7fa2a69408, n= 10000000
sorted 2 of 100000000 in 4 seconds
begin= 0x7f7fa76b4810, n= 10000000
sorted 3 of 100000000 in 6 seconds
begin= 0x7f7fac2ffc18, n= 10000000
sorted 4 of 100000000 in 8 seconds
begin= 0x7f7fb0f4b020, n= 10000000
sorted 5 of 100000000 in 9 seconds
begin= 0x7f7fb5b96428, n= 10000000
sorted 6 of 100000000 in 11 seconds
begin= 0x7f7fba7e1830, n= 10000000
sorted 7 of 100000000 in 13 seconds
begin= 0x7f7fbf42cc38, n= 10000000
sorted 8 of 100000000 in 14 seconds
begin= 0x7f7fc4078040, n= 10000000
sorted 9 of 100000000 in 16 seconds
begin= 0x7f7fc8cc3448, n= 10000000
sorted 10 of 100000000 in 19 seconds

As the algorithm works its way through the billion elements (100 million
at a time), N remains constant but T grows rather dramatically. I
suspect that's because of I/O (and maybe I should measure that, too).

Suggestions? What would you do?

--jkl

[Brett Lymn]

Post by James K. Lowden
Suggestions? What would you do?

I would use msync(2) with the MS_INVALIDATE flag on your test file to
invalidate the cached pages.

[James K. Lowden]

On Tue, 12 Aug 2014 12:15:38 +0930

Post by Brett Lymn

Post by James K. Lowden
Suggestions? What would you do?

I would use msync(2) with the MS_INVALIDATE flag on your test file to
invalidate the cached pages.

Thanks, Brett. I'm afraid that's a fine answer to the question I
asked, instead of the one I should have asked....

Each test is a separate process, and tests are executed sequentially.
My concern is that the kernel's pagemap cache (or whatever it's called)
will still contain file pages from the input file when process N+1
runs, and that the kernel will, rationally, take advantage of that
fact to make mmap(2) as fast as possible.

AIUI, msync(2) on process N wouldn't affect mmap(2) on process N+1.
If the last thing process N does is

msync(fd, len, MS_INVALIDATE);
exit(0);

then when process N+1 calls mmap on the same file, the kernel is
certainly positioned to know whether or not those pages have in fact
been changed (and, in this case, they have not). So there's no
technical *need* for the kernel to expend I/O to re-read pages it
"knows" haven't changed. Does it do that anyway?

I thought I would need a kernel sysctl or something, a nonportable knob
to force re-reading the file from disk. If such exists.

--jkl

[Brett Lymn]

Post by James K. Lowden
On Tue, 12 Aug 2014 12:15:38 +0930

Post by Brett Lymn

Post by James K. Lowden
Suggestions? What would you do?

I would use msync(2) with the MS_INVALIDATE flag on your test file to
invalidate the cached pages.

Thanks, Brett. I'm afraid that's a fine answer to the question I
asked, instead of the one I should have asked....
Each test is a separate process, and tests are executed sequentially.
My concern is that the kernel's pagemap cache (or whatever it's called)
will still contain file pages from the input file when process N+1
runs, and that the kernel will, rationally, take advantage of that
fact to make mmap(2) as fast as possible.

I believe the pages are marked as invalid in the page cache which means
they will be pulled back in from disk on the next access. I was using
MS_INVALIDATE on executable files, I had a small tool that would mmap
and invalidate the pages. The next time the target executable file ran
the pages definitely came back from disk. So, yes, if the pages are
shared then run N+1 would re-fetch the page.

[David Holland]

Post by James K. Lowden
I'm testing different algorithms for performance against a single
memory-mapped file. How can I best ensure that for each run the
page cache is cold, that the prior run left nothing for the current run
to take advantage of?

The safest way is (has always been, for all benchmarking) to reboot
between runs.

[Joerg Sonnenberger]

Post by James K. Lowden
I'm testing different algorithms for performance against a single
memory-mapped file. How can I best ensure that for each run the
page cache is cold, that the prior run left nothing for the current run
to take advantage of?

Try to put the file on a separate file system and unmount/mount it?
That should flush the associated pages.

Joerg