changed 4 months ago  
Published Linked with GitHub

pmairca

  • need dev to run this full time
  • hand to tres

deepwriter

  • thesis integration and grant

resource devs

society library

  • make an announcement

-> ruby
-> toji

-> send simon scripts

  • intros

  • engineering – incentivizing engineers continnuously, project manager / paymaster

  • all hands dev

  • agents.fun

  • marc trading / talking

  • dev team project manager

  • video – how actions work? and create an example action

  • video – how buying and selling works

GHFUFU9wCAo5wNHAYpuFimCWZVcKB9cWPZq61vTv8gsN

talk to

bags for people

  • nuro
  • marvin
  • amiller?

420.69 degenai

GLK7vPS4s2qwNFtYcLBGgvDvBrV82LGwG73E8Z9J8yW

Dp3sGsTxqaJCeXLnKCByk5xE2JH8jT8F4jn24eFEUwz

FkF5UkpFKqRxMFZwyP8tNDWuspxRti5f4RT8fZYnXnJQ

B7CVJcUGGUB5wf93qYfoCqzaKLWd9rb8GsCHBaQkgJe9
(ai16z)

DV3GwhvVNdxDo4QbC3kuvDPp64GFeXbfjW8m4Px2idAD

E33ZdtjgcRWU1jq4gPUK6f4A9AuRbHkAeWBzw52xK7nj
(ai16z)

8umiQsP9ruUQKYJaH1uGAdeFWLJGM22op5NLMKQJvMDi

SmNFzSFWjw1D7MupayiiPDKqJ7ofrAKhwhUgFD9e97c
(ai16z)

B9vsqgi4k18R1U57vbPfmszzL4iiXrAuQgbLzHdFnAif

8yMMCsvy8GLx28eqto5hPCKLd1ffzpPHdUKNX8HUVM9N
(ai16z)

DozxqfkebDwJX5JpvktPYYKFRDYPsRnjxRoCU26s63i9
(ai16z)

CSyQiUW24akEeCMFrkEunaFh4r9wqyMrq1RpHPo2XRjn
(ai16z)

BijRpZ8m8hsskZCswoVSijFsTHymUnpGNizMLoRE6UWk
(ai16z)

67KyJ32h915Rb8n41FK9Fxqa9gZceBen51dG7TkRASJB

77UJXoS2ZdxKhRSrsskc2VNZNxUGRRc49YAedEeXgeN9
(ai16z)

RATkHXSC3vdBAr3JGT4r41WKeTpqp7ShMLsj7YLmuic

CoYVSz1XhdPtnbpCs52BXhH9Vd5Jj87imTkcnuSxzerG
(ai16z)

6GBMvVmRpi5ULHAksXmh2r42cYtumyVeNK42xsTNUZsU
(ai16z)

6g8Ht698cftQJ398ukyeJNF6nUT9SZGhPMHeRXd7cnBJ
(ai16z)

B7CVJcUGGUB5wf93qYfoCqzaKLWd9rb8GsCHBaQkgJe9
(ai16z)

DV3GwhvVNdxDo4QbC3kuvDPp64GFeXbfjW8m4Px2idAD

B9vsqgi4k18R1U57vbPfmszzL4iiXrAuQgbLzHdFnAif

8umiQsP9ruUQKYJaH1uGAdeFWLJGM22op5NLMKQJvMDi

4jayUCGpvD55oRZjrSytnMvtj9ZtX7Mn4fQocDcubYmu
(ai16z)

6GfrJ3Bx3prRPi24ZHZLEi55vgZkmsg1ot4gNttBAvRw

88AcL51SgtssC61UuyeVbEFYSLtZX2iJzXNpUKFnw2Ly

8yMMCsvy8GLx28eqto5hPCKLd1ffzpPHdUKNX8HUVM9N
(ai16z)

A9wgvLnnKCAxLGrKUY6DfLm17bPZhzMSATWtuBEyuxfW

2PPAFeMFXkxZnNWLz49JZum6wGF6CCGjLozoarorTK1g (ai16z)

6aNqjFLPiSbM6AL9M6Fi9gdWXEbMsgUvDiupS5QM1ktG
(ai16z)

32BXHTbQd1PRBGu6pLV33uwY4zVJriv7gUGTgaiBtZG6
(ai16z)

C6LXeG3cai7akTta9gUCVVaaMpsoH7aaj6DSoj4h7vwC

2nis2NCbtSM66ePRhHzkfrX2svxvNgpS8WzavSDqpedw
(ai16z)

ai16z
0x5390B04839C6EBaA765886E1EDe2BCE7116e462F

JC4MuEFnegNfyhKpw98qeDjtonYLUY7cpUvr4ZYzFeFQ

FpoEDUYoYaBzQykJVmLcXTJjmVJ3dzYeXx4eRNAw1VoZ

8XeG9qPWu3GebXPYybARAnmRFt2xJbBnDykLb2zNNDMu

B4G7sRjbJUQs1VGpMt9USgfGmuDmDU9xGv6wcKmN1GRA

8KwwcFVcTuitGx7SBGiykC6bRCEtHKGig6RHpKhTpLw6

4W94nPVv8kkfW1vUSRXFYdGFzeDfcFJkzddSAdQxN5nj

4Y2vPARZfVyanK1pTrKkzkrFKac1RB2rDGdzfRKT2RkP

4Mq5tV3xoJMKcSPBRbjnDaMjxvjmnYFnB8GZzPEZaVB6

BmhyHiqK2gfDsUwtxwobgxTS1Xm674w5D7CP53B6fowC

2QPjLYsRkF9wrdscRtVLjAtgaNy2PF6FFheYSDMsy4zE

Deoji.sol

9e5GHLdPyW65y9L2pnuRq1w1MdGq5PFL3WoZ5XsaeHvS

CYT3pFiKbUwVL3DfQxs5UfuwiwCnPJ5wZbYnAfucyWAP

8amyfjQ2fzFJ3ibTxMk8zerYXCa9AKSQdoTKGTjucq1u

2AKhrRD8nZKSTvTS9AbKvsxwcHHnca1fy3AaKZs7hydc

BvCYv78uZqmzf2hhoAR6ahRfRgcqwdD9fXWcGT3Wcoxt

6cb86eUJsPZg1JKnmxXrgTYR2yBKamc5cu69sQDqDwrm

H77fmHepthmffu8sXB1Sc8QvQhK9fvygqURCXyPLsG3K

5cEG9GpfeiMzvbHWVRua5EzmZ5LEFQvKsPfLLKkLruhA

0x405530f066FedC505f30c3460D4De300f16bF074

AHZ4Kfr9qy6GLn46SFmTHQHFY8Xim7sRfY4gWKFH1Lc6

C58R7FQFGdYgMiRwHLEsnyYCUpaUxj6AyYLZxS9DdmyK

8N4J92AenuNcEWvVu7S3CmzbxPytNkFbS6Ytyc6fU11b
degenai
4bhTQs3C8bdiS6ancFftsD9LnwTPyAHUQfRf8JfmX6WC

7BP7UWAR6JnzMyFyPc4UUK7W8zGSBJKr6eZQv7RTv5Xk

25an5PGePSiRYsp8toHaBKwFJDdRnwy69aZwwtki6fbH

B7w6NQtbfyS8ZodYWVi8tMQrh4gseLxnWtyn1sAkHfzp

6HgzpqSGZSZJmpjuHUg4JCELMi4VRpPcYtqChHG7uYz

EfRjzw1YxjJLsBKCwPyLTxruVWyhdSFZxp32VszoKwo7

CkwWi9x8GscLiPjGPGkLN8wS752t3DtbGBBnVNwy7yAk

HG7qvvHW2bbW3eWGoHKdQT6qr6vgZj2QQ5Nhvhc9Nhet

0x7f5971D611E81A80d552DE7ffFcC85e3d35Cf182

Gs71jtmhsu31CpVjEe6GZt7Qgt4qhiu6Cb3cS5JieWvY

23zgam4wGeeCeHHHrXuSNEX8LdMjkN3nwzZLE442hk55

7ZCRzFTNREyotT6YTtH7yQ2RA1MnijPGZL2RifkJw1Zp

68wryxQJ7BZWUainU82iUyzx6JNF8JNFRxdaoAXtMpg5

5J2dQdWsvpH9YyMUjUfkBtqTsNennNDKLfxR46ZCAEGC

HjcZ4HdWXYSRFCwwcxSmKCJbL2YT78aKuMWMJpjWjnmw

-give each token

  • fix brains between ruby and spartan

  • make marc trade

  • fix dev keys

  • fix leaky rooms

https://huggingface.co/datasets/MasaFoundation/huberman_lab_Marc_Andreessen_How_Risk_Taking_Innovation__Artificial_Intelligence_Transform_Human

  • Generate with FAL.AI
  • Fine tune

Degen Spartan

  • Fix voice

  • Add wallet credentials

  • Test wallet balance

form

replace \n\n with spaces

remove cringe emojis

track goat and gnon and other top dex prices
awareness of btc price, eth price
swaps, burns transfers, balances

evaluators for alpha

SMOL MVP

  • when clicking on chat, dont hit the canvas
  • when typing in chat, dont do the hotkeys
  • chat to selected agent

onboard alex

degen's tweets
train fine tune model
make a couch

  • Movement

  • Actions and interactions

  • Entity description and interactions (drink coffee, play games on computer, etc)

  • Personality Matrix System + Eval loop

  • Relationships

SMOL 1.0

  • World editor
  • Multiplayer

Basic interactions

Next

  • local embedding memory storage and handling with indexeddb
  • Asset loader / preloader
  • User interface menus and options
  • Lifecycle
  • Agent-agent interactions
  • Add additional entity types

Later

  • Local model option / testing

  • Pay-to-run chatgpt

  • Gestures

  • World editor

  • Market

  • Inventory

  • Design

  • Wearables

  • on chain agent memories

Multi-Agent Simulation Game
-> User comes, logs in with Twitter
-> Agent is generated from their profile
-> User can tell their agent what to do – no direct interaction
-> All agents run persistently
-> Hard problems – how to run a persistent multi-agent simulation for very cheap?

-> Every agent is assigned a role in the story

-> Long term

Naptha cares more about things like– a stock market simulator. They would love the 3D thing or Twitter / social graph thing.

GDD update

team splits

Plan austin

  1. EPRC Wiki
  • Smol Integration

TODO:

  1. BIA

-> put pfp image and name on canvas
-> make sure the share button works with twitter auth etc
-> move callbacks to the cloud version

  1. SMOLS

SELECTION SYSTEM

-> click to select
-> check entity, current selected
-> show seletion UI
-> inspect button to bring up modal

1. MOVE TO

add move goto action -> solve for going in front of item
-> Actions have targets

Entities have descriptions and actions, actions show for any nearby entities

If agent calls entity action, moves to entity and then does it if not already next to it

  1. speech bubble for character responses
  2. show last message sent by user at bottom

make up with friends now we not stressed

0. data-driven refactor

-> get smols and worlds in chain state
-> get and store layout - do we want a serialized grid or ASCII? 8 bit in each one and worlds have small medium large massive
-> add create world
-> add create smol
-> show modal for changing smol world

-> list monkeys in UI

make monkeys selectable

2. GESTURES

add gesture to response, separate from action
GESTURE_WAVE
GESTURE_BELLYRUB
GESTURE_ADMIRE
GESTURE_NOD
GESTURE_CLAP
GESTURE_IMPATIENCE

for now, just log the gestures with a stub playGesture async function, we'll need a gesture list

fix message examples and add gestures throughout the examples

-> if last message from from last person is continue, agent should not continue since they may be following up

  1. Select monkey
    -> view all chats in all rooms with monkey as collapsibles
  2. Add personality scores
  3. Add evaluator for scoring

Select objects?

  1. deploy frontend and proxy with CI/CD
  2. set up MUD deployment on testnet

Memory system
-> Instead of sqlite, rewrite for indexeddb
-> dont write embeddings
-> store on chain with no embeddings
->

UI SYSTEM

-> Log In
-> Select world

animation triggers from emojis in message?

  1. clean up ui

  2. user settings

  3. cohort

  4. add characters to cohort

  5. refresh cohort

  6. Generate character from…
    -> twitter archive .zip
    -> tweets.js
    -> twitter url

  7. upload / download character

  8. upload / download cohort

REST FRAMEWORK
-> crud cohort
-> crud characters

M3 / Shaw

  1. add knowledge search and injection

  2. add action for ruby to consume an attachment as knowledge
    Send tweets from Discord
    -> requires admin knowledge

  3. Create a moon

  4. Add jin, nature, etc

RUBY

tweets need to make sure not respond to already responded to

post – respond directly to, not just about. it feels like we're just riffing, ah yes.

add more detail to ruby profile

twitter messages are kind of boring we should think about the post goals

  1. should respond handler for voice / better interest handling

conversation mode or just tell eliza to ignore

4. twitter action and evaluation handling on interaction and search

5. twitter actions

twitter actions handler - twitter needs processactions
7. mute thread (should be working already, just need to make sure actions are handled)
8. like message
9. respond
10. retweet
11. rewteet with comment
12. follow
13. ignore

eliza needs to be able to end a conversation and not always continue – esp if thye user is challenging or asking eliza to make pipe bomb, interaction should ignore

what whould eliza do?
-> don't respond / ignore
-> reply to the thread
-> retweet the post with a reply
-> like the post

  1. Add twitter actions
  2. multi-agent conversation needs to not suck

Error: Cannot perform IP discovery - socket closed
at Socket.<anonymous> (file:///home/jin/repo/Eliza/node_modules/@discordjs/voice/dist/index.mjs:309:46)
at Object.onceWrapper (node:events:634:28)
at Socket.emit (node:events:532:35)
at socketCloseNT (node:dgram:768:8)
at process.processTicksAndRejections (node:internal/process/task_queues:89:21)
Emitted 'error' event on VoiceConnection instance at:
at VoiceConnection.onNetworkingError (file:///home/jin/repo/Eliza/node_modules/@discordjs/voice/dist/index.mjs:1861:10)
at Networking.emit (node:events:520:28)
at file:///home/jin/repo/Eliza/node_modules/@discordjs/voice/dist/index.mjs:676:32
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

  1. character file from documents

  2. users make character file

msf archive

summaries as attachments?
-> should have "short" version that is added after attachment

role-based context provider

finding attachments / previous posts

More fixes

Discord -> Handle replies, show (in reply to ID) for messages, if message is replying to ruby this is an automatic response usually

-> wrong voice / dm context
Eliza is currently in a direct message conversation with moon
Eliza is not currently in a voice channel

3. post actions / content response

If the agent only likes the post, this will signify an ironic acknolwedgement without having to say anything
If the agent feels attacked or has nothing positive to say, better to just not respond

-> should respond and ignore

-> like message
-> like message instead – if we like a message instead of responding, it indicates a very human level of irony

interaction fixes

sleep cycle for posting/messaging/responding
-> if interaction interaction handler has a lot of messages to catch up on, filter down somehow
-> rate limit interaction responses - interaction responses should be much slower on twitter

discord sendTyping

  1. test llama local summarization handling
  2. check on our token limits so we effectively use our token amts
  3. handle local llama voice generation with different voices per user
    review and make sure all prompts right – interaction showing "unknown user"

SPRINT 3

document chunking and vector search
chat with pdf, perplexity trick

TWITTER_MODE = active or passive, set in character
twitter DMs obv

blocking / unblocking user on discord
show (BLOCKED)

send friend

blocking/unblocking user on twitter

imagecompletion is a runtime thing

make sure that all discord messages are also rate limited, re sending and receiving

tests pass

-> better temp and data file handling, store in ~/tmp or Application Support or whatever
-> script that creats and gets the folder path for temp files everywhere

we can build a bio / profile on users from their twitter account as we interact with them

handle actors, user bios, etc

back populate channels where ruby as invited for context / populate past discord messages

discord - handle edits?

Introductions and cojourney stuff?

Evaluation and profile building stuff?

  1. possibly too aggressive about ignoring, but still asking too many questions

  2. if we can detect cuda and amount of gpu memory, we can use gemma2b, llama3 8b1b

If agent leaves voice room, should say by first then leave

if we start getting the person speaking, interrupt

filter join voice if nobody is in any voice channel

modes - local_low, hybrid_low, cloud_low, local_med, local_high, cloud_high

-> do we want to burn down the queue if we're overrunning on completions? or throw warning if queue is > 10

smart relationsihp graphing / building knowledge on a person

dynamic local model downloading

-> can we weight down retrieval of older facts?

twitter_to_discord provider

discord_to_twitter provider

twitter_to_discord action

discord_to_twitter action
-> follow
-> retweet
-> etc

goals?

joining and leaving voice chat should leave messages in voice channels
-> respond to users joining
dont respond to users leaving

MUTE_ROOM, like IGNORE, should not respond

llama message completion, handle case where afterjson

extract basic information from people like where they live etc

can we mix cheap and expensive? responses with gpt-4-mini but factuals, whispers,

  1. handle video watching and other long running tasks by acknowledging, then watch, then follow up

get all prior interactions with the user if they exist (useful for context)

summarize channel, summarize other channel (must be linked or named, like join voice)
Action? Summarize channel, summarize other channel, summarize voice channel

SPRINT 3

generate actions
-> search yco and post about it
-> search arxiv and post about it
-> search wikipedia and post about it
-> search google and post about it

twitter spaces!!

-> point to your twitter, import

SPRINT 3

npx rubyagent and CLI to run an agent locally

SPRINT 4

web interface for logging

Eliza console script

First time user

The first time the user starts the script, check if there are any characters in ~/.eliza/characters

The user can create a new character or select an existing character. The user can select multiple characters by pressing down or up to select, then pressing ENTER to toggle selection on the character. Press "c" to continue once at least one character has been selected. If the user presses "e" on a character it will walk them through the prompts to edit again.

For character generation ->

  1. Create manually – go through all
  2. Generate from prompt
  3. Import twitter .zip

If "Create manually" is selected, start with these prompts:

  1. Character name -> if already character with same name, throw error and make user try again
  2. Character bio
  3. Character "lore" – user can and, delete and edit lore items, which are each strings in an array, then continue when they have finished adding lore
  4. Message examples – ask 5 questions and have the user respond in character
  5. Descriptive adjectives
  6. Relevant topics the agent is interested in
  7. Style directions for chat
    -> all, chat, post

Generate prompt and import twitter zip should respond with "not implemented" and go back

-> generate lore
-> generate character
-> create character from twitter .zip

  1. add user_id to goals

goals should have a user_id if they dont
sort goals by actor in prompt

  1. fix tests
    {
    "testName": "Test multiple elaborate messages in a conversation",
    "attempts": 3,
    "successful": 0,
    "successRate": 0
    },
    {
    "testName": "Test if message is added to database",
    "attempts": 3,
    "successful": 0,
    "successRate": 0
    },
    {
    "testName": "Extract programmer and startup facts",
    "attempts": 3,
    "successful": 0,
    "successRate": 0
    },

FAIL src/lib/tests/lore.test.ts
● Console

console.log
Room found for participants 9ff62aa3-28f0-4635-ae2d-9f0b2c60cb6c

at getOrCreateRelationship (test/getOrCreateRelationship.ts:37:13)

console.error
Error adding lore Error: {"code":"23503","details":"Key (room_id)=(00000000-0000-0000-0000-000000000000) is not present in table "rooms".","hint":null,"message":"insert or update on table "memories" violates foreign key constraint "reflections_room_id_fkey""}
at SupabaseDatabaseAdapter.createMemory (/Users/shawwalters/bgent-new/src/lib/adapters/supabase.ts:271:15)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at MemoryManager.createMemory (/Users/shawwalters/bgent-new/src/lib/memory.ts:150:5)
at addLore (/Users/shawwalters/bgent-new/src/lib/lore.ts:40:5)
at Object.<anonymous> (/Users/shawwalters/bgent-new/src/lib/tests/lore.test.ts:67:7)

45 | });
46 | } catch (e) {

47 | console.error("Error adding lore", e);
| ^
48 | throw e;
49 | }
50 | }

at addLore (lib/lore.ts:47:13)
at Object.<anonymous> (lib/tests/lore.test.ts:67:7)

console.error
Error: {"code":"23503","details":"Key (room_id)=(00000000-0000-0000-0000-000000000000) is not present in table "rooms".","hint":null,"message":"insert or update on table "memories" violates foreign key constraint "reflections_room_id_fkey""}
at SupabaseDatabaseAdapter.createMemory (/Users/shawwalters/bgent-new/src/lib/adapters/supabase.ts:271:15)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at MemoryManager.createMemory (/Users/shawwalters/bgent-new/src/lib/memory.ts:150:5)
at addLore (/Users/shawwalters/bgent-new/src/lib/lore.ts:40:5)
at Object.<anonymous> (/Users/shawwalters/bgent-new/src/lib/tests/lore.test.ts:67:7)

78 | });
79 | } catch (error) {

80 | console.error(error);
| ^
81 | }
82 | expect(lore[0].content).toEqual(content);
83 | }, 60000);

at Object.<anonymous> (lib/tests/lore.test.ts:80:15)

console.error
Error adding lore Error: {"code":"23503","details":"Key (room_id)=(00000000-0000-0000-0000-000000000000) is not present in table "rooms".","hint":null,"message":"insert or update on table "memories" violates foreign key constraint "reflections_room_id_fkey""}
at SupabaseDatabaseAdapter.createMemory (/Users/shawwalters/bgent-new/src/lib/adapters/supabase.ts:271:15)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at MemoryManager.createMemory (/Users/shawwalters/bgent-new/src/lib/memory.ts:150:5)
at addLore (/Users/shawwalters/bgent-new/src/lib/lore.ts:40:5)
at Object.<anonymous> (/Users/shawwalters/bgent-new/src/lib/tests/lore.test.ts:88:5)

Make sure all tests clean up successfully

  1. make sure extra participants and users arent added, might be more than 1…

  2. updateGoals - successfully updates an existing goal

  3. introduce validation check – check for goals not finished

  4. Add introduce provider which gives introduce state (cannot introduce, need more data, can)

  5. add ftu goal test

  6. try to get all the way through FTU and finish goal

  7. credits, userdata

Tutorial Videos

Matchmaking CRON?

fn_notify_agents not triggered

​​​​Later SL stuff
​​​​
​​​​2. review and de-deduplicate works, authors, publishers, etc
​​​​
​​​​2. Some arxiv and biorxiv requests fail (maybe files were removed for some reason?) but are available on archive.org, so we could resolve them

-> google scholar is blocking our request
-> resolve missing years with archive.org

  1. fix twitter
    -> determine which tweets are relevant to the research, if any
    -> crawl links
    -> return data as a post

  2. Add cookies and caching for handling youtube download in worker

  3. Google scholar can get blocked by captcha, solve it with proxy service

  4. Deploy worker to SL and Publish lib

  5. fix crawl issues in google scholar? (not rate limiting since fucked)

  6. fix issues with semantic scholar? seems to get stuck?? timeout too long?

  7. Add fade out to splash page

  8. authors and institutions are not well linked to each other

  9. black text background and font?

I got these notes from an engineer who suggested this is how I could optimize my code

Spatial optimization of the graph
break entire sim into 4 x 4 x 4 (64 blocks)

place every node into a block based on position
if camera switches blocks it is in recompute, recompute visible blocks
visible blocks are all adjacent blocks to block camera is in, or all adjacent to the nearest to camera if camera is outside of the simulation spae
if a block is disabled, all of the nodes and links inside the block are disabled
if a block is enabled, all of the nodes and links inside the block are enabled
For debug, render the active blocks with a green wireframe / outline, and the inactive blocks with a red wireframe / outline

Please review, write out a whole plan in plain english. Then, propose a pseudo-code architecture for all of this. Then, review your approach, and identify any flaws in it. Finally, write out your finally solution, writing all new functions and code complete with no omissions

Blog post

if institutions is empty, hide on author

  1. restyle

connecting authors to institutions better?

  1. add label backgounds which are same blue

link authors across papers?

filterbar could group my works, authors, publications, institutions as the main point, and its one or the other

  1. record fly-through

authors
-> get the name of the instution for each entry instead of whole object
[{'id': '4c141fcd-ff12-5513-a7d1-4f107d025f05', 'type': 'institution', 'source_type': 'openalex', 'institution_type': 'academic', 'name': 'Technical University of Munich', 'url': None, 'description': None}]
-> drop publications

institutions
-> drop url, description, publications

publications
-> drop description, institutions, doi, publisher

w

  1. Make prompt that shows topic subtopic list and makes model output list of topics with

  2. output cleaner -> remove full text, remove new lines and styling of json, remove source_type, links, filter by relevant and remove relevant

  3. export CSV for works, authors, instititutions, conferences

finalize

  1. deliver final json of all scrape
  2. finalize crawl and add final json
Select a repo