Add optimized parser with multithreading, SIMD, and custom allocator support (squashed take-over of #424) by syoyo · Pull Request #428 · tinyobjloader/tinyobjloader

syoyo · 2026-05-21T19:16:07Z

Take-over of #424 (branch copilot/optimize-parser-for-huge-meshes, originally by the Copilot coding agent).

CI did not run on #424 because it was opened by a bot, and the branch carried 31 incremental commits. This PR re-applies the identical final tree as a single squashed commit on top of current release, pushed by a maintainer so the Unit/Python workflows trigger normally.

What's included

New optimized OBJ loader (LoadObjOpt / LoadObjOptTyped) targeting huge meshes, with fast_float-based ASCII float/double parsing.
Optional compile-time features:
- TINYOBJLOADER_USE_MULTITHREADING — threaded parse + merge
- TINYOBJLOADER_USE_SIMD — SSE2/AVX2/NEON newline scanning
- TINYOBJLOADER_ENABLE_EXCEPTION — opt-in C++ exceptions
ArenaAllocator / arena_adapter for bulk allocation and a TypedArray API.
Experimental stream-based OBJ loader under experimental/stream/.
Fuzzing harnesses (tests/obj-fuzz, tests/llvm-fuzz, tests/fuzz_common.h) and extensive parity tests against the legacy loader (tests/opt/loadobjopt_multithread.inc, tester.cc).

Verification

Squashed tree is byte-identical to Add optimized parser with multithreading, SIMD, and custom allocator support #424 head (614cca0) for all 18 changed files.
cd tests && make check passes locally under clang++ -std=c++11 -fsanitize=address (60/60 unit tests OK).

Closes #424.

🤖 Generated with Claude Code

…support Squashed take-over of #424 (branch copilot/optimize-parser-for-huge-meshes) so CI runs on a maintainer-pushed branch. Highlights: - New optimized OBJ loader (LoadObjOpt / LoadObjOptTyped) targeting huge meshes, with fast_float-based ASCII float/double parsing. - Optional compile-time features: TINYOBJLOADER_USE_MULTITHREADING (threaded parse + merge), TINYOBJLOADER_USE_SIMD (SSE2/AVX2/NEON newline scanning), and TINYOBJLOADER_ENABLE_EXCEPTION (opt-in C++ exceptions). - ArenaAllocator / arena_adapter for bulk allocation and a TypedArray API. - Experimental stream-based OBJ loader under experimental/stream/. - Fuzzing harnesses (tests/obj-fuzz, tests/llvm-fuzz, tests/fuzz_common.h) and extensive parity tests against the legacy loader (tests/opt/loadobjopt_multithread.inc, tester.cc). Original work by Copilot coding agent; reviewed and validated locally (make check passes under clang++ with -std=c++11 -fsanitize=address). Co-Authored-By: Copilot <198982749+Copilot@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR brings in a new high-performance OBJ parsing path (optimized whole-buffer loader + experimental stream loader) and expands the test/fuzzing infrastructure to validate parity with the legacy loader and improve robustness for large/complex inputs.

Changes:

Added extensive unit/parity tests for the optimized loader APIs (LoadObjOpt, LoadObjOptTyped) including multithreaded correctness checks.
Introduced an experimental stream-based OBJ loader (experimental/stream) plus differential fuzzing harnesses (tests/obj-fuzz, tests/llvm-fuzz).
Updated test build scripts/docs to build the new components and expose fuzz targets.

Reviewed changes

Copilot reviewed 16 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/tester.cc	Enables multithreaded build for tests and adds many new unit/parity tests for optimized + typed APIs.
tests/README.md	Documents additional `make obj-fuzz` / `make llvm-fuzz` targets.
tests/opt/loadobjopt_multithread.inc	Adds a large suite of parity and multithread determinism tests for optimized and stream loaders.
tests/obj-fuzz/README.md	Documents the structured differential fuzzer and how to run it.
tests/obj-fuzz/obj_fuzz.cc	Implements a deterministic structured OBJ generator and cross-check runner.
tests/obj-fuzz/Makefile	Build/run rules for the structured fuzzer.
tests/Makefile	Links the stream loader into `tester` and adds convenience targets/clean steps for fuzzers.
tests/llvm-fuzz/README.md	Documents the libFuzzer harness and intended cross-check scope.
tests/llvm-fuzz/Makefile	Build/run rules for the libFuzzer harness.
tests/llvm-fuzz/fuzz_loaders.cc	libFuzzer entry point exercising multiple parser paths with optional cross-check asserts.
tests/llvm-fuzz/corpus/seed_basic.obj	Adds a seed corpus OBJ to bootstrap fuzzing.
tests/fuzz_common.h	Shared helpers for splitting fuzz inputs and comparing parse results across loaders.
loader_example.cc	Updates the example material reader to use C++11 `override`.
experimental/stream/stream_obj_loader.h	Declares the experimental stream loader API and handler interface.
experimental/stream/stream_obj_loader.cc	Implements the experimental stream loader (chunked parsing + ordered replay).
experimental/stream/README.md	Documents scope/goals/limitations of the stream parser.
experimental/README.md	Mentions the new stream parser entry point under `experimental/`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    if (tokens.size() >= 4) {
+      real_t maybe_r = real_t(1.0);
+      if (ParseRealToken(tokens[3], &maybe_r)) {
+        if (tokens.size() == 4) {
+          event.has_vertex_weight = true;
+          event.vertex_weight = maybe_r;
+        } else {
+          real_t maybe_g = real_t(1.0);
+          if (!ParseRealToken(tokens[4], &maybe_g)) {
+            event.has_vertex_weight = true;
+            event.vertex_weight = maybe_r;
+          } else if (tokens.size() >= 6) {
+            real_t maybe_b = real_t(1.0);
+            if (ParseRealToken(tokens[5], &maybe_b)) {
+              event.has_vertex_weight = true;
+              event.vertex_weight = maybe_r;
+              event.has_color = true;
+              event.r = maybe_r;
+              event.g = maybe_g;
+              event.b = maybe_b;
+            }
+          }
+        }
+      }


+void test_arena_adapter_overflow_guard() {
+  // Verify that arena_adapter::allocate rejects SIZE_MAX/sizeof(T) overflow.
+  // When TINYOBJLOADER_ENABLE_EXCEPTION is not defined, the allocator returns
+  // nullptr on overflow.  When exceptions are enabled, it throws std::bad_alloc.
+  tinyobj::ArenaAllocator arena;
+  tinyobj::arena_adapter<double> adapter(&arena);
+  // Request an allocation that would overflow size_t when multiplied by
+  // sizeof(double)=8.  SIZE_MAX / 8 + 1 overflows.
+  double *p = adapter.allocate(SIZE_MAX / sizeof(double) + 1);
+  TEST_CHECK(p == nullptr);


+namespace tinyobj {
+namespace experimental_stream {
+struct StreamLoadConfig {
+  bool triangulate;
+  bool default_vcols_fallback;
+  int num_threads;
+  size_t chunk_line_count;
+
+  StreamLoadConfig()
+      : triangulate(true),
+        default_vcols_fallback(false),
+        num_threads(1),
+        chunk_line_count(4096) {}
+};
+
+bool LoadObjStreamExperimental(
+    attrib_t *attrib, std::vector<shape_t> *shapes,
+    std::vector<material_t> *materials, std::string *warn, std::string *err,
+    std::istream *input, MaterialReader *readMatFn,
+    const StreamLoadConfig &config = StreamLoadConfig());
+
+bool LoadObjStreamExperimental(
+    attrib_t *attrib, std::vector<shape_t> *shapes,
+    std::vector<material_t> *materials, std::string *warn, std::string *err,
+    const char *filename, const char *mtl_basedir,
+    const StreamLoadConfig &config = StreamLoadConfig());
+}  // namespace experimental_stream
+}  // namespace tinyobj


Address the three review comments from Copilot on the take-over PR, plus a related robustness fix uncovered while doing so. - experimental/stream: fix `v x y z w r g b` (7-component) vertex parsing. The stream loader previously treated the weight `w` as red and shifted the colors. It now mirrors LoadObjOpt, keyed on the component count beyond xyz: +1 -> weight, +3 -> color (weight = r, legacy compat), +4 -> weight + color. - tests/tester.cc: include the canonical experimental/stream/stream_obj_loader.h instead of re-declaring StreamLoadConfig / LoadObjStreamExperimental, so the test cannot drift from the real API. - tests/tester.cc: make test_arena_adapter_overflow_guard conditional on TINYOBJLOADER_ENABLE_EXCEPTION (allocate() throws std::bad_alloc when exceptions are enabled, returns nullptr otherwise) so the test no longer terminates when built with exceptions. - tiny_obj_loader.h: guard the IMPLEMENTATION block against double inclusion within a single translation unit (TINYOBJLOADER_IMPLEMENTATION_DEFINED). Including the stream header after defining TINYOBJLOADER_IMPLEMENTATION now pulls in tiny_obj_loader.h a second time; without the guard this caused redefinition errors. - Add test_stream_loader_weighted_color_vertex_7_components regression test. Verified: `make check` passes under clang++ and g++, both with and without -DTINYOBJLOADER_ENABLE_EXCEPTION. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

syoyo · 2026-05-21T19:47:35Z

Addressed @copilot's review in ed77593:

stream_obj_loader.cc — 7-component v x y z w r g b: the stream loader treated w as red and shifted colors. Rewrote the v weight/color handling to mirror LoadObjOpt, keyed on the component count beyond x y z: +1 → weight, +3 → color (weight = r, legacy compat), +4+ → weight + color. Added regression test test_stream_loader_weighted_color_vertex_7_components.
tester.cc — arena overflow test: made test_arena_adapter_overflow_guard conditional on TINYOBJLOADER_ENABLE_EXCEPTION — it now catches std::bad_alloc when exceptions are enabled and expects nullptr otherwise, so it no longer terminates in an exception-enabled build.
tester.cc — duplicated API decls: now #includes the canonical experimental/stream/stream_obj_loader.h instead of re-declaring StreamLoadConfig / LoadObjStreamExperimental, so the test can't silently drift from the real header.

Follow-on hardening that #3 surfaced: including the stream header after TINYOBJLOADER_IMPLEMENTATION is defined pulls tiny_obj_loader.h into the TU a second time, which redefined every implementation symbol. Added a TINYOBJLOADER_IMPLEMENTATION_DEFINED guard around the implementation block.

Verified make check passes under both clang++ and g++, with and without -DTINYOBJLOADER_ENABLE_EXCEPTION.

…sion Web research against the official Wavefront spec, Paul Bourke's vertex-color reference, and Wikipedia confirms there is no standard 7-component `v` line: the format defines only xyz / xyzw / xyzrgb, and weight and color are mutually exclusive. Per maintainer decision, keep the weight + color interpretation (4th value = weight, next three = RGB) in LoadObjOpt and the experimental stream loader, but document clearly that it is a tinyobjloader-specific extension and that the classic LoadObj / LoadObjWithCallback path instead caps at 6 components, so the loaders intentionally diverge for 7+ token vertices. Comment-only change; no behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…d code Ensure the library's default configuration (no SIMD, no multithreading, no exceptions) is the baseline that CI exercises, and clean up warnings/dead code introduced by the optimized parser. Default configuration: - tester.cc no longer force-defines TINYOBJLOADER_USE_MULTITHREADING, so the default `make check` build now runs with the library defaults (scalar, single-threaded, exception-free). The header already had no auto-#define of any feature macro, so consumers were already defaulting to all-off; this just makes the test build match. - tests/Makefile builds two binaries: `tester` (defaults) and `tester_features` (TINYOBJLOADER_USE_MULTITHREADING + _USE_SIMD + ENABLE_EXCEPTION). `make check` runs both, so CI covers the default and feature-enabled code paths. New `check_default` / `check_features` targets run a single config. - Document the opt-in macros (all disabled by default) in the header and tests/README. Polish / hardening: - Remove dead function opt_requires_legacy_fallback (a never-called duplicate of opt_line_start_is_legacy). - opt_tryParseDouble: drop the dead inner fast_float branch (unreachable in every configuration) and guard the whole function with TINYOBJLOADER_DISABLE_FAST_FLOAT, since it is only called from that fallback path. Removes two -Wunused-function warnings in the default build. - Guard the now-unused `cache` parameter in opt_parseLineToThreadData under TINYOBJLOADER_DISABLE_FAST_FLOAT (-Wunused-parameter). Verified: default, features, and TINYOBJLOADER_DISABLE_FAST_FLOAT builds all compile (clang++ and g++, C++11 and C++17) and pass the full suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

`make check` now builds and runs a third configuration, `check_nofastfloat`, which defines TINYOBJLOADER_DISABLE_FAST_FLOAT to exercise the hand-written fallback float parser (otherwise library defaults). CI invokes `make check`, so this fallback path — previously only verified by hand — is now covered automatically alongside the default and feature-enabled builds. Verified: all three configs (default, features, nofastfloat) build and pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replace the stale "Optimized loader" section (which referenced an old experimental loader and v0.9/v1.0 benchmark numbers) with a short usage guide for the in-header LoadObjOpt API: a compilable example, the opt-in multithreading/SIMD macros (off by default), and a note on the arena-backed LoadObjOptTyped variant and the experimental stream loader. Also add a "What's new" entry. Example verified to compile (-Wall -Wextra) and run against models/cornell_box.obj. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings May 21, 2026 19:16

Copilot started reviewing on behalf of syoyo May 21, 2026 19:16 View session

Copilot AI reviewed May 21, 2026

View reviewed changes

Copilot started work on behalf of syoyo May 21, 2026 19:47 View session

Copilot finished work on behalf of syoyo May 21, 2026 19:48

syoyo and others added 4 commits May 22, 2026 04:57

syoyo merged commit 62ff207 into release May 21, 2026
30 checks passed

syoyo deleted the optimize-parser-squashed branch May 21, 2026 21:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add optimized parser with multithreading, SIMD, and custom allocator support (squashed take-over of #424)#428

Add optimized parser with multithreading, SIMD, and custom allocator support (squashed take-over of #424)#428
syoyo merged 6 commits into
releasefrom
optimize-parser-squashed

syoyo commented May 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

syoyo commented May 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

syoyo commented May 21, 2026

What's included

Verification

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

syoyo commented May 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants