Update README.md

.github/workflows/ccpp.yml

@@ -38,6 +38,7 @@ jobs:
         g++ --version
         nvcc --version
         make VERBOSE=1
+
   build:
     runs-on: ubuntu-latest
     steps:
@@ -61,3 +62,6 @@ jobs:
         cd build
         g++ --version
         make VERBOSE=1
+    - name: test
+      run: |
+        build/tools/perfect-cli -h

README.md

@@ -17,6 +17,7 @@ CPU/GPU Performance control library for benchmarking on Linux, x86, POWER, and N
 - [x] CUDA not required (GPU functions will not be compiled)
 - [x] Flush file system caches (linux)
 - [x] Disable ASLR (linux)
+- [x] process priority interface (linux)
 
 ## Contributors
 * [Carl Pearson](https://cwpearson.github.io)
@@ -61,6 +62,8 @@ g++ code_using_perfect.cpp -I perfect/include
 
 ## Tools Usage
 
+### tools/perfect-cli
+
 `perfect` provides some useful tools on Linux:
 
 ```
@@ -87,9 +90,9 @@ The basic usage is `tools/perfect-cli -- my-exe`, which will attempt to configur
 Most modifications require elevated privileges.
 The default behavior is to:
 * disable ASLR
-* drop filesystem caches
 * set CPU performance to maximum
 * disable CPU turbo
+* drop filesystem caches before each iteration
 
 Some options (all should provided before the `--` option):
 * `--no-mod` flag will cause `perfect-cli` to not modify the system performance state
@@ -97,6 +100,45 @@ Some options (all should provided before the `--` option):
 * `--stderr`/`--stdout` will redirect the program-under-test's stderr and stdout to the provided paths.
 * `-s`/`-u`: set the number of shielded /unshielded CPUs. The program-under-test will run on the shielded CPUs. All other tasks will run on the unshielded CPUs.
 
+A common invocation might look like:
+```
+sudo tools/perfect-cli -n 5 --stderr=run.err --stdout=run.out -- ./my-benchmark
+```
+This will disable ASLR, set CPU performance to maximum, disable CPU turbo, and then run `./my-benchmark` 5 times after dropping the filesystem cache before each run, redirecting stdout/stderr of ./my-benchmark to `run.out`/`run.err`.
+The owner of `run.out` and `run.err` will be set to whichever user called `sudo`.
+
+### tools/addr
+
+Print the address of `main`, a stack variable, and a heap variable.
+Useful for demoing ASLR.
+
+### tools/no-aslr
+
+Disable ASLR on the provided execution.
+
+With ASLR, addresses are different with each invocation
+```
+$ tools/addr
+main:  94685074364704
+stack: 140734279743492
+heap:  94685084978800
+$ tools/addr
+main:  93891046344992
+stack: 140722671706708
+heap:  93891068624496
+```
+
+Without ASLR, addresses are the same in each invocation
+```
+$ tools/no-aslr tools/addrs       
+main:  93824992233760
+stack: 140737488347460
+heap:  93824994414192
+$ tools/no-aslr tools/addrs       
+main:  93824992233760
+stack: 140737488347460
+heap:  93824994414192
+```
 
 ## API Usage
 
@@ -109,7 +151,19 @@ perfect::CpuTurboState state;
 PERFECT(perfect::get_cpu_turbo_state(&state));
 ```
 
-## Monitoring
+### High Priority
+
+`perfect` can set high scheduling priority for a process
+
+See [examples/high_priority.cpp](examples/high_priority.cpp)
+
+```c++
+#include "perfect/priority.hpp"
+```
+
+* `Result set_high_priority()`: set the highest possible scheduling priority for the calling process
+
+### Monitoring
 
 `perfect` can monitor and record GPU activity.
 
@@ -140,10 +194,6 @@ See [tools/no_aslr.cpp](tools/no_aslr.cpp)
 * `Result set_aslr(const AslrState &state)`: set a previously-saved ASLR state
 
 
-
-
-
-
 ### Flush file system caches
 
 `perfect` can drop various filesystem caches
@@ -232,40 +282,6 @@ See [examples/cpu_cache.cpp](examples/cpu_cache.cpp).
 
 * `void flush_all(void *p, const size_t n)`: Flush all cache lines starting at `p` for `n` bytes.
 
-## Tools
-
-### tools/addr
-
-Print the address of `main`, a stack variable, and a heap variable.
-Useful for demoing ASLR.
-
-### tools/no-aslr
-
-Disable ASLR on the provided execution.
-
-With ASLR, addresses are different with each invocation
-```
-$ tools/addr
-main:  94685074364704
-stack: 140734279743492
-heap:  94685084978800
-$ tools/addr
-main:  93891046344992
-stack: 140722671706708
-heap:  93891068624496
-```
-
-Without ASLR, addresses are the same in each invocation
-```
-$ tools/no-aslr tools/addrs       
-main:  93824992233760
-stack: 140737488347460
-heap:  93824994414192
-$ tools/no-aslr tools/addrs       
-main:  93824992233760
-stack: 140737488347460
-heap:  93824994414192
-```
 
 ## Changelog
 
@@ -296,21 +312,17 @@ heap:  93824994414192
 
 - [ ] only monitor certain GPUs
 - [ ] hyperthreading interface
-- [ ] process priority interface
+
-- [ ] A wrapper utility
-    - [ ] disable hyperthreading
-    - [ ] reserve cores 
-    - [ ] set process priority
-    - [ ] disable ASLR
 
 ## Related
 
 * [LLVM benchmarking instructions](https://llvm.org/docs/Benchmarking.html#linux) covering ASLR, Linux governor, cpuset shielding, SMT, and Intel turbo.
-* [easyperf.net](https://easyperf.net/blog/2019/08/02/Perf-measurement-environment-on-Linux#2-disable-hyper-threading) blog post discussing ACPI/Intel turbo, SMT, Linux governor, CPU affinity, process priority, file system caches, and ASLR. 
+* [easyperf.net blog post](https://easyperf.net/blog/2019/08/02/Perf-measurement-environment-on-Linux#2-disable-hyper-threading) discussing ACPI/Intel turbo, SMT, Linux governor, CPU affinity, process priority, file system caches, and ASLR. 
-* [temci](https://github.com/parttimenerd/temci) benchmarking tool for cpu sheilding and disabling hyperthreading, among other things.
+* [parttimenerd/temci](https://github.com/parttimenerd/temci) benchmarking tool for cpu sheilding and disabling hyperthreading, among other things.
-* [perflock](https://github.com/aclements/perflock) tool for locking CPU frequency scaling domains
+* [aclements/perflock](https://github.com/aclements/perflock) tool for locking CPU frequency scaling domains
+* [lpechacek/cpuset](https://github.com/lpechacek/cpuset) python package/tool for managing CPU shielding
 
 ## Acks
 
-Uses [muellan/clipp](https://github.com/muellan/clipp) for cli option parsing.
+* Uses [muellan/clipp](https://github.com/muellan/clipp) for cli option parsing.
-Uses [martinmoene/optional-lite](https://github.com/martinmoene/optional-lite).
+* Uses [martinmoene/optional-lite](https://github.com/martinmoene/optional-lite).

examples/CMakeLists.txt

@@ -43,6 +43,9 @@ target_link_libraries(cpu-turbo perfect)
 add_executable(os-perf os_perf.cpp)
 target_link_libraries(os-perf perfect)
 
+add_executable(high-priority high_priority.cpp)
+target_link_libraries(high-priority perfect)
+
 if(CMAKE_CUDA_COMPILER)
     add_executable(gpu-clocks gpu_clocks.cu)
     target_link_libraries(gpu-clocks perfect)

examples/high_priority.cpp

@@ -0,0 +1,12 @@
+#include <iostream>
+
+#include "perfect/priority.hpp"
+
+int main(void) {
+  perfect::init();
+
+  PERFECT(perfect::set_high_priority());
+
+  // do things with high process scheduling priority
+
+}

include/perfect/cpu_set.hpp

@@ -12,18 +12,10 @@
 #include <string>
 #include <vector>
 
+#include "detail/fs.hpp"
 #include "init.hpp"
 #include "result.hpp"
 
-#define SUCCESS_OR_RETURN(stmt) \
-{\
-  Result _ret; \
-  _ret = (stmt); \
-if (_ret != Result::SUCCESS) {\
-  return _ret;\
-}\
-}
-
 std::set<int> operator-(const std::set<int> &lhs, const std::set<int> &rhs) {
   std::set<int> result;
   for (auto e : lhs) {
@@ -34,6 +26,17 @@ std::set<int> operator-(const std::set<int> &lhs, const std::set<int> &rhs) {
   return result;
 }
 
+// intersection
+std::set<int> operator&(const std::set<int> &lhs, const std::set<int> &rhs) {
+  std::set<int> result;
+  for (auto e : lhs) {
+    if (1 == rhs.count(e)) {
+      result.insert(e);
+    }
+  }
+  return result;
+}
+
 std::string remove_space(const std::string &s) {
   std::string result;
 
@@ -86,7 +89,6 @@ std::set<int> parse_token(const std::string &token) {
 }
 
 std::set<int> parse_cpuset(const std::string &s) {
-  // std::cerr << "parse_cpuset: parsing '" << s << "'\n";
   std::set<int> result;
 
   std::string token;
@@ -109,11 +111,12 @@ namespace perfect {
 class CpuSet {
 public:
   std::string path_;
-  std::set<int> cpus_;
-  std::set<int> mems_;
   CpuSet *parent_;
 
-  // make sure cpuset is initialized
+  CpuSet() : path_(""), parent_(nullptr) {}
+  CpuSet(const CpuSet &other) : path_(other.path_), parent_(other.parent_) {}
+
+  // make sure cpuset system is initialized
   static Result init() {
 
     // check for "nodev cpuset" in /proc/filesystems
@@ -162,23 +165,24 @@ public:
     return Result::SUCCESS;
   }
 
-  std::string get_raw_cpus() {
+  std::string get_raw_cpus() const {
-    std::ifstream is(path_ + "/cpuset.cpus");
+    std::string path = path_ + "/cpuset.cpus";
+    std::ifstream is(path);
     std::stringstream ss;
     ss << is.rdbuf();
     return remove_space(ss.str());
   }
 
-  std::string get_raw_mems() {
+  std::string get_raw_mems() const {
     std::ifstream is(path_ + "/cpuset.mems");
     std::stringstream ss;
     ss << is.rdbuf();
     return remove_space(ss.str());
   }
 
-  std::set<int> get_cpus() { return parse_cpuset(get_raw_cpus()); }
+  std::set<int> get_cpus() const { return parse_cpuset(get_raw_cpus()); }
 
-  std::set<int> get_mems() { return parse_cpuset(get_raw_mems()); }
+  std::set<int> get_mems() const { return parse_cpuset(get_raw_mems()); }
 
   // migrate the caller task from this cpu set to another
   Result migrate_self_to(CpuSet &other) {
@@ -193,11 +197,12 @@ public:
     std::string line;
     while (std::getline(is, line)) {
       line = remove_space(line);
+
       if (std::to_string(self) == line) {
-        // std::cerr << "migrating self task " << line << " to " << other.path
+        // std::cerr << "migrating self task " << line << " to " << other.path_
         //            << "\n";
-        other.write_task(line);
+        pid_t pid = std::stoi(line);
-        return Result::SUCCESS;
+        return other.write_task(pid);
       }
     }
     return Result::NO_TASK;
@@ -205,46 +210,58 @@ public:
 
   // migrate tasks in this cpu set to another
   Result migrate_tasks_to(CpuSet &other) {
+    // other must have cpus and mems
+    auto s = other.get_cpus();
+
+    assert(!other.get_cpus().empty());
+    assert(!other.get_mems().empty());
+
     // enable memory migration in other
-    SUCCESS_OR_RETURN(other.enable_memory_migration());
+    PERFECT_SUCCESS_OR_RETURN(other.enable_memory_migration());
 
     // read this tasks and write each line to other.tasks
     std::ifstream is(path_ + "/tasks");
     std::string line;
     while (std::getline(is, line)) {
-      // std::cerr << "migrating task " << line << " to " << other.path << "\n";
+      pid_t pid = std::stoi(line);
-      other.write_task(line);
+      // std::cerr << "migrating task " << pid << " to " << other.path_ << "\n";
+      Result result = other.write_task(pid);
+      if (Result::ERRNO_INVALID == result) {
+        // std::cerr << "task " << pid << " is unmovable\n";
+      } else {
+        PERFECT_SUCCESS_OR_RETURN(result);
+      }
     }
-
     return Result::SUCCESS;
   }
 
   Result enable_memory_migration() {
-    std::ofstream ofs(path_ + "/" + "cpuset.memory_migrate");
+    return detail::write_str(path_ + "/cpuset.memory_migrate", "1");
-    ofs << "1";
-    ofs.close();
-    if (ofs.fail()) {
-    switch (errno) {
-    case EACCES:
-      return Result::NO_PERMISSION;
-    case ENOENT:
-      return Result::NOT_SUPPORTED;
-    default:
-      return Result::UNKNOWN;
-    }
-  }
-    return Result::SUCCESS;
   }
 
-  void write_task(const std::string &task) {
+  Result write_task(pid_t pid) {
-    // write `task` to path/tasks
+    return detail::write_str(path_ + "/tasks", std::to_string(pid) + "\n");
-    std::ofstream os(path_ + "/tasks");
+  }
-    os << task << "\n";
+
+  static Result get_affinity(std::set<int> &cpus, pid_t pid) {
+    cpu_set_t mask;
+    CPU_ZERO(&mask);
+    if (sched_getaffinity(pid, sizeof(mask), &mask)) {
+      return from_errno(errno);
+    }
+
+    cpus.clear();
+    for (int i = 0; i < CPU_SETSIZE; ++i) {
+      if
+        CPU_ISSET(i, &mask) { cpus.insert(i); }
+    }
+
+    return Result::SUCCESS;
   }
 
   // object representing the root CPU set
   static Result get_root(CpuSet &root) {
-    SUCCESS_OR_RETURN(CpuSet::init());
+    PERFECT_SUCCESS_OR_RETURN(CpuSet::init());
     root.path_ = "/dev/cpuset";
     root.parent_ = nullptr;
     return Result::SUCCESS;
@@ -256,7 +273,7 @@ public:
   Result make_child(CpuSet &child, const std::string &name) {
 
     if (mkdir((path_ + "/" + name).c_str(),
-              S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH)) {
+              S_IRUSR | S_IWUSR | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH)) {
       switch (errno) {
       case EEXIST: {
         // okay
@@ -264,8 +281,6 @@ public:
       }
       case EACCES:
         return Result::NO_PERMISSION;
-      case ENOENT:
-      case EROFS:
       default:
         return Result::UNKNOWN;
       }
@@ -276,6 +291,8 @@ public:
     return Result::SUCCESS;
   }
 
+  std::vector<CpuSet> get_children() { assert(false && "unimplemented"); }
+
   Result enable_cpu(const int cpu) {
     std::set<int> cpus = get_cpus();
     cpus.insert(cpu);
@@ -290,30 +307,28 @@ public:
     return write_cpus(finalCpus);
   }
 
-  // FIXME: check error
   Result write_cpus(std::set<int> cpus) {
-    std::ofstream os(path_ + "/cpuset.cpus");
+    std::string str;
     bool comma = false;
     for (auto cpu : cpus) {
       if (comma)
-        os << ",";
+        str += ",";
-      os << cpu << "-" << cpu;
+      str += std::to_string(cpu) + "-" + std::to_string(cpu);
       comma = true;
     }
-    return Result::SUCCESS;
+    return detail::write_str(path_ + "/cpuset.cpus", str);
   }
 
-  // FIXME: check write
   Result write_mems(std::set<int> mems) {
-    std::ofstream os(path_ + "/cpuset.mems");
+    std::string str;
     bool comma = false;
     for (auto mem : mems) {
       if (comma)
-        os << ",";
+        str += ",";
-      os << mem << "-" << mem;
+      str += std::to_string(mem) + "-" + std::to_string(mem);
       comma = true;
     }
-    return Result::SUCCESS;
+    return detail::write_str(path_ + "/cpuset.mems", str);
   }
 
   Result enable_mem(const int mem) {
@@ -331,31 +346,40 @@ public:
   }
 
   Result destroy() {
+
+    // already destroyed
+    if (!detail::path_exists(path_)) {
+      return Result::SUCCESS;
+    }
+
     // remove all child cpu sets
 
     // move all attached processes back to parent
-    assert(parent_);
+    assert(parent_ && "should not call destroy on root cpuset");
-    migrate_tasks_to(*parent_);
+    PERFECT_SUCCESS_OR_RETURN(migrate_tasks_to(*parent_));
 
     // remove with rmdir
+    Result result = Result::UNKNOWN;
     if (rmdir(path_.c_str())) {
       switch (errno) {
+      case ENOENT:
+        // already gone
+        result = Result::SUCCESS;
+        break;
       default:
         std::cerr << "unhandled error in rmdir: " << strerror(errno) << "\n";
-        return Result::UNKNOWN;
+        result = Result::UNKNOWN;
       }
     }
 
     path_ = "";
-    return Result::SUCCESS;
+    return result;
   }
-
-
 };
 
-  std::ostream &operator<<(std::ostream &s, const CpuSet &c) {
+std::ostream &operator<<(std::ostream &s, const CpuSet &c) {
   s << c.path_;
   return s;
-  }
+}
 
 } // namespace perfect

include/perfect/detail/fs.hpp

@@ -1,5 +1,6 @@
 #pragma once
 
+#include <cstring>
 #include <fstream>
 #include <string>
 
@@ -32,15 +33,20 @@ Result write_str(const std::string &path, const std::string &val) {
   if (ofs.fail()) {
     switch (errno) {
     case EACCES:
-    std::cerr << "EACCES when writing to " << path << "\n";
+      // std::cerr << "EACCES when writing to " << path << "\n";
       return Result::NO_PERMISSION;
     case EPERM:
-      std::cerr << "EPERM when writing to " << path << "\n";
+      // std::cerr << "EPERM when writing to " << path << "\n";
       return Result::NO_PERMISSION;
     case ENOENT:
-    std::cerr << "ENOENT when writing to " << path << "\n";
+      // std::cerr << "ENOENT when writing to " << path << "\n";
       return Result::NOT_SUPPORTED;
+    case EINVAL:
+      // std::cerr << "EINVAL when writing to " << path << "\n";
+      return Result::ERRNO_INVALID;
     default:
+      std::cerr << strerror(errno) << " when writing " << val << " to " << path
+                << "\n";
       return Result::UNKNOWN;
     }
   }

include/perfect/detail/os/linux.hpp

@@ -13,6 +13,8 @@
 #include <sys/types.h>
 #include <unistd.h>
 #include <sys/personality.h>
+#include <sys/time.h>
+#include <sys/resource.h>
 
 #include "perfect/result.hpp"
 
@@ -107,6 +109,26 @@ Result set_personality(const int persona) {
   }
   return Result::SUCCESS;
 }
+
+// give the calling process the highest priority
+Result set_high_priority() {
+  if (setpriority(PRIO_PROCESS, 0, -20)) {
+    return from_errno(errno);
+  }
+  return Result::SUCCESS;
 }
 
+// disable all but one SMT thread for all CPUs the calling process can run on
+Result disable_smt() {
+  return Result::NOT_SUPPORTED;
+}
+
+// enable SMT for all CPUs the calling process can run on
+Result enable_smt() {
+return Result::NOT_SUPPORTED;
+}
+
+
+} // namespace detail
+
 } // namespace perfect

include/perfect/priority.hpp

@@ -0,0 +1,15 @@
+#pragma once
+
+#ifdef __linux__
+#include "detail/os/linux.hpp"
+#else
+#error "unsupported platform"
+#endif
+
+#include "init.hpp"
+
+namespace perfect {
+    Result set_high_priority() {
+        return detail::set_high_priority();
+    }
+}

include/perfect/result.hpp

@@ -22,7 +22,7 @@ enum class Result {
   NO_PERMISSION,
   NOT_SUPPORTED,
   NO_TASK,
-
+  ERRNO_INVALID,
   NVML_NO_PERMISSION,
   NVML_NOT_SUPPORTED,
   NVML_UNINITIALIZED,
@@ -88,6 +88,8 @@ const char *get_string(const Result &result) {
     return "unsupported operation";
   case Result::NO_TASK:
     return "no such task";
+  case Result::ERRNO_INVALID:
+    return "errno EINVAL";
   case Result::UNKNOWN:
     return "unknown error";
   case Result::NVML_NOT_SUPPORTED:
@@ -118,10 +120,10 @@ inline void check(Result result, const char *file, const int line) {
 #define PERFECT(stmt) check(stmt, __FILE__, __LINE__);
 
 #define PERFECT_SUCCESS_OR_RETURN(stmt)                                        \
-{\
+  {                                                                            \
     Result _ret;                                                               \
     _ret = (stmt);                                                             \
-if (_ret != Result::SUCCESS) {\
+    if (_ret != Result::SUCCESS) {                                             \
-  return _ret;\
+      return _ret;                                                             \
-}\
+    }                                                                          \
-}
+  }

tools/migrate-to-cpuset.sh

@@ -0,0 +1,14 @@
+#! /bin/bash
+
+while read i; do
+  echo $i;
+  echo $i > /dev/cpuset/tasks;
+done < /dev/cpuset/unshielded/tasks
+
+while read i; do
+  echo $i;
+  echo $i > /dev/cpuset/tasks;
+done < /dev/cpuset/shielded/tasks
+
+rmdir /dev/cpuset/shielded
+rmdir /dev/cpuset/unshielded

tools/perfect.cpp

@@ -1,12 +1,16 @@
 #include <cassert>
 #include <cerrno>
+#include <chrono>
+#include <functional>
 #include <iostream>
 #include <string>
+#include <thread>
 #include <vector>
 
 #ifdef __linux__
 #include <fcntl.h>
 #include <pwd.h>
+#include <signal.h>
 #include <sys/stat.h>
 #include <sys/types.h>
 #include <sys/wait.h>
@@ -24,6 +28,27 @@
 #include "perfect/detail/os/linux.hpp"
 #include "perfect/drop_caches.hpp"
 #include "perfect/os_perf.hpp"
+#include "perfect/priority.hpp"
+
+typedef std::function<perfect::Result()> CleanupFn;
+std::vector<CleanupFn> cleanups;
+
+// restore the system state to how we found it
+void cleanup(int dummy) {
+  (void)dummy;
+  std::cerr << "caught ctrl-c\n";
+
+  // unregister our handler
+  signal(SIGINT, SIG_DFL);
+  std::cerr << "cleaning up\n";
+  std::cerr << "ctrl-c again to quit\n";
+
+  for (auto f : cleanups) {
+    perfect::Result result = f();
+  }
+
+  exit(EXIT_FAILURE);
+}
 
 // argv should be null-terminated
 // outf and errf are file descriptors to where stdout and stderr should be
@@ -142,6 +167,9 @@ int fork_child(char *const *argv, int outf, int errf) {
 }
 
 int main(int argc, char **argv) {
+
+  signal(SIGINT, cleanup);
+
   using namespace clipp;
 
   size_t numUnshielded = 0;
@@ -150,11 +178,17 @@ int main(int argc, char **argv) {
   nonstd::optional<bool> cpuTurbo = false;
   nonstd::optional<bool> maxOsPerf = true;
   bool dropCaches = true;
+  bool highPriority = true;
 
   std::vector<std::string> program;
   std::string stdoutPath;
   std::string stderrPath;
   int iters = 1;
+  int sleepMs = 1000;
+
+  bool help = false;
+
+  auto helpMode = option("-h", "--help").set(help).doc("show help");
 
   auto shieldGroup = ((option("-u").doc("number of unshielded CPUs") &
                        value("INT", numUnshielded)) |
@@ -166,7 +200,8 @@ int main(int argc, char **argv) {
                         .set(aslr, true)
                         .call([&]() { cpuTurbo = nonstd::nullopt; })
                         .call([&]() { maxOsPerf = nonstd::nullopt; })
-                        .set(dropCaches, false));
+                        .set(dropCaches, false)
+                        .set(highPriority, false));
 
   auto modMode = (shieldGroup,
                   option("--no-drop-cache")
@@ -176,6 +211,9 @@ int main(int argc, char **argv) {
                     maxOsPerf = false;
                   }),
                   option("--aslr").set(aslr, true).doc("enable ASLR"),
+                  option("--no-priority")
+                      .set(highPriority, false)
+                      .doc("don't set high priority"),
                   option("--cpu-turbo").doc("enable CPU turbo").call([&]() {
                     cpuTurbo = true;
                   }),
@@ -184,8 +222,11 @@ int main(int argc, char **argv) {
                   (option("--stderr").doc("redirect child stderr") &
                    value("PATH", stderrPath)));
 
-  auto cli = ((noModMode | modMode),
+  auto cli =
-              (option("-n").doc("run multiple times") & value("INT", iters)),
+      helpMode |
+      ((noModMode | modMode),
+       (option("--sleep-ms").doc("sleep before run") & value("INT", sleepMs)),
+       (option("-n").doc("run multiple times") & value("INT", iters)), helpMode,
        // run everything after "--"
        required("--") & greedy(values("cmd", program))
 
@@ -197,6 +238,12 @@ int main(int argc, char **argv) {
     return -1;
   }
 
+  if (help) {
+    auto fmt = doc_formatting{}.doc_column(31);
+    std::cout << make_man_page(cli, argv[0], fmt);
+    return 0;
+  }
+
   // open the redirect files, if needed
   int errf = 0;
   int outf = 0;
@@ -245,13 +292,14 @@ int main(int argc, char **argv) {
     }
   }
 
-  // exec the rest of the options
+  // build the program arguments
   std::vector<char *> args;
   for (auto &c : program) {
     args.push_back((char *)c.c_str());
   }
   args.push_back(nullptr);
 
+  // init the perfect library
   PERFECT(perfect::init());
 
   auto cpus = perfect::cpus();
@@ -262,34 +310,44 @@ int main(int argc, char **argv) {
   }
 
   // handle CPU shielding
-  perfect::CpuSet shielded, unshielded;
+  perfect::CpuSet root, shielded, unshielded;
   if (numShielded) {
     std::cerr << "shielding " << numShielded << " cpus\n";
 
-    perfect::CpuSet root;
     PERFECT(perfect::CpuSet::get_root(root));
     PERFECT(root.make_child(shielded, "shielded"));
     PERFECT(root.make_child(unshielded, "unshielded"));
 
     std::cerr << "enable memory\n";
     PERFECT(shielded.enable_mem(0));
-    PERFECT(shielded.enable_mem(0));
+    PERFECT(unshielded.enable_mem(0));
 
     std::cerr << "enable cpus\n";
     size_t i = 0;
-    for (; i < numShielded; ++i) {
+    for (; i < cpus.size() - numShielded; ++i) {
-      std::cerr << "shield cpu " << cpus[i] << "\n";
-      shielded.enable_cpu(cpus[i]);
-    }
-    for (; i < cpus.size(); ++i) {
       std::cerr << "unshield cpu " << cpus[i] << "\n";
       unshielded.enable_cpu(cpus[i]);
     }
+    for (; i < cpus.size(); ++i) {
+      std::cerr << "shield cpu " << cpus[i] << "\n";
+      shielded.enable_cpu(cpus[i]);
+    }
 
     std::cerr << "migrate self\n";
     PERFECT(root.migrate_self_to(shielded));
-    std::cerr << "migrate other\n";
+    std::cerr << "migrate other (1/2)\n";
     PERFECT(root.migrate_tasks_to(unshielded));
+    // some tasks may have been spawned by unmigrated tasks while we migrated
+    std::cerr << "migrate other (2/2)\n";
+    PERFECT(root.migrate_tasks_to(unshielded));
+
+    cleanups.push_back(CleanupFn([&] {
+      std::cerr << "cleanup: shielded cpu set\n";
+      shielded.destroy();
+      std::cerr << "cleanup: unshielded cpu set\n";
+      unshielded.destroy();
+      return perfect::Result::SUCCESS;
+    }));
   }
 
   // handle aslr
@@ -309,6 +367,11 @@ int main(int argc, char **argv) {
       std::cerr << "enabling cpu turbo\n";
       PERFECT(perfect::enable_cpu_turbo());
     }
+
+    cleanups.push_back(CleanupFn([&] {
+      std::cerr << "cleanup: restore CPU turbo state\n";
+      return perfect::set_cpu_turbo_state(cpuTurboState);
+    }));
   }
 
   // handle governor
@@ -321,21 +384,39 @@ int main(int argc, char **argv) {
         PERFECT(perfect::os_perf_state_maximum(cpu));
       }
     }
+
+    cleanups.push_back(CleanupFn([&] {
+      std::cerr << "cleanup: os governor\n";
+      return perfect::set_os_perf_state(osPerfState);
+    }));
   }
 
-  // handle file system caches
+  if (highPriority) {
+    std::cerr << "set high priority\n";
+    PERFECT(perfect::set_high_priority());
+  }
+
+  // parent should return
+  for (int runIter = 0; runIter < iters; ++runIter) {
+
+    // drop filesystem caches before each run
     if (dropCaches) {
       std::cerr << "clearing file system cache\n";
       PERFECT(perfect::drop_caches());
     }
 
-  // parent should return
+    // sleep before each run
-  for (int runIter = 0; runIter < iters; ++runIter) {
+    if (sleepMs) {
+      std::cerr << "sleep " << sleepMs << " ms before run\n";
+      std::this_thread::sleep_for(std::chrono::milliseconds(sleepMs));
+    }
+
     std::cerr << "exec ";
     for (size_t i = 0; i < args.size() - 1; ++i) {
       std::cerr << args[i] << " ";
     }
     std::cerr << "\n";
+
     int status = fork_child(args.data(), outf, errf);
     if (0 != status) {
       std::cerr << "did not terminate successfully\n";