OOM while fetch big site#13

OOM when I fetch a big site, like https://docs.llamaindex.ai/en/stable/ version: sitemcp/0.3.3 darwin-arm64 node-v22.14.0 command: sitemcp https://docs.llamaindex.ai/en/stable/ -t pathname --concurrency 3 the error is : ``` <--- Last few GCs ---> [36005:0x148008000] 281965 ms: Mark-Compact 3996.8 (4129.5) -> 3995.2 (4129.5) MB, pooled: 0 MB, 3136.96 / 0.00 ms (average mu = 0.302, current mu = 0.064) allocation failure; scavenge might not succeed [36005:0x148008000] 286734 ms: Mark-Compact 4015.5 (4133.3) -> 4000.8 (4149.5) MB, pooled: 3 MB, 4616.17 / 0.00 ms (average mu = 0.143, current mu = 0.032) allocation failure; scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ----- Native stack trace ----- 1: 0x1049571e4 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 2: 0x104b56fd8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 3: 0x104d64f30 v8::internal::Heap::stack() [/mypath/.nvm/versions/node/v22.14.0/bin/node] 4: 0x104d632d0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 5: 0x104d578b8 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 6: 0x104d580f0 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 7: 0x104d3b410 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 8: 0x105156ae4 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 9: 0x105619af4 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/mypath/.nvm/versions/node/v22.14.0/bin/node] 10: 0x105665bbc Builtins_CreateEmptyArrayLiteral [/mypath/.nvm/versions/node/v22.14.0/bin/node] 11: 0x10b56fc14 12: 0x10b56ff5c 13: 0x10b56ff5c 14: 0x10b56ff5c 15: 0x10b56ff5c 16: 0x10b54d6d4 17: 0x10b772ef4 18: 0x10568e6c0 Builtins_ProxyGetProperty [/mypath/.nvm/versions/node/v22.14.0/bin/node] 19: 0x10563485c Builtins_ArrayForEach [/mypath/.nvm/versions/node/v22.14.0/bin/node] 20: 0x10b9af450 21: 0x10b59b40c 22: 0x10b55c220 23: 0x10b50d9a8 24: 0x1055c1290 Builtins_AsyncFunctionAwaitResolveClosure [/mypath/.nvm/versions/node/v22.14.0/bin/node] 25: 0x10568c4d8 Builtins_PromiseFulfillReactionJob [/mypath/.nvm/versions/node/v22.14.0/bin/node] 26: 0x1055b1594 Builtins_RunMicrotasks [/mypath/.nvm/versions/node/v22.14.0/bin/node] 27: 0x105582af4 Builtins_JSRunMicrotasksEntry [/mypath/.nvm/versions/node/v22.14.0/bin/node] 28: 0x104cbce6c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 29: 0x104cbd714 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 30: 0x104cbd84c v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 31: 0x104ceaac8 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 32: 0x104ceb24c v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 33: 0x1048619a8 node::InternalCallbackScope::Close() [/mypath/.nvm/versions/node/v22.14.0/bin/node] 34: 0x104861474 node::InternalCallbackScope::~InternalCallbackScope() [/mypath/.nvm/versions/node/v22.14.0/bin/node] 35: 0x1048eb83c node::Environment::RunTimers(uv_timer_s*) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 36: 0x10555d220 uv__run_timers [/mypath/.nvm/versions/node/v22.14.0/bin/node] 37: 0x1055609f4 uv_run [/mypath/.nvm/versions/node/v22.14.0/bin/node] 38: 0x104862508 node::SpinEventLoopInternal(node::Environment*) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 39: 0x1049a0154 node::NodeMainInstance::Run() [/mypath/.nvm/versions/node/v22.14.0/bin/node] 40: 0x104916364 node::Start(int, char**) [/mypath/.nvm/versions/node/v22.14.0/bin/node] 41: 0x183f5eb4c start [/usr/lib/dyld] Abort trap: 6 ```