wc (GNU coreutils clone in Rust)

A high-performance clone of GNU wc written in Rust that matches or exceeds GNU wc performance while adhering strictly to its behavior, output formatting, flag combinations, and edge-case handling.

Features & Optimizations

• Vectorized Line Counting (-l): AVX-512 / AVX2 / NEON hardware acceleration processing up to 256 bytes per loop unroll with single-cycle POPCNT bitmask counting.
• Fast-Path Bytes Counting (-c): Instant fs::metadata lookup for regular files with fallback for /proc and /sys virtual files.
• Fast Chars Counting (-m): Vectorized lead-byte / non-continuation byte checking (b & 0xC0) != 0x80.
• High-Throughput Word Counting (-w): SIMD bit-parallel state scanner for ASCII chunks with UTF-8 decoding fallback.
• Visual Column Line Length (-L): Tab expansion to 8-column tabstops, printable ASCII fast-path with memchr3, and unicode-width visual display calculation.
• Memory-Mapped I/O: Regular files exceeding 64 KiB are mapped with memmap2 and advised with Advice::Sequential.
• Parallel Processing with Rayon: Multi-file inputs are automatically processed in parallel across worker threads while preserving original argument order.
• Strict Formatting & Flags: Complete parity with GNU wc column width calculation (compute_number_width), --total=WHEN, --files0-from=F, and shell quoting for filenames containing newlines.

Building & Testing

# Build in release mode
cargo build --release

# Run unit and integration tests
cargo test --release

Running Benchmarks

An extensive, automated benchmarking script is provided in benches/benchmark.sh:

./benches/benchmark.sh

Benchmarking External Implementations

The benchmark script supports adding arbitrary external implementations (e.g. uutils coreutils, BusyBox, BSD wc, or alternative Rust utilities):

# Compare against uutils and busybox
./benches/benchmark.sh --add "uutils:/path/to/uutils-wc" --add "busybox:busybox wc"

# Choose a different baseline for relative speedup calculations
./benches/benchmark.sh --add "other:/path/to/wc" --baseline "other"

# Run only a specific test suite with custom iterations
./benches/benchmark.sh --suite ascii --runs 10

# Pass external tools via environment variable
EXTRA_WC="uutils:uutils-wc custom:/path/to/wc" ./benches/benchmark.sh

Benchmark Scope

1. 100 MB ASCII Text File: Evaluates SIMD kernels, memory-mapping, and branchless scanners across all single and combined flags.
2. 50 MB Multilingual UTF-8 Text File: Evaluates multibyte decoding throughput across English, Chinese, French, Japanese, Greek, and emojis.
3. Piped Stdin (Streaming I/O): Tests buffered reader throughput through Unix pipes.
4. 500 Files Concurrency: Benchmarks multi-file parallel processing using Rayon vs GNU wc sequential processing.
5. Correctness Verification: Verifies byte-for-byte output equivalence against the baseline before each benchmark.