[FYI] We disable the "Create New Article" function at present

[shaoyijia]

Thanks everyone for trying our demo and providing feedback!

Due to very high demand, we have to disable the "Create New Article" function at present since our search API has exceeded its maximum limit in less than two days (#12). You can still see examples on the web demo or opt in for email notification when we bring the function back.

Also, as we are researching on automatic content curation, supporting customized sources is on our roadmap. PRs on integrating other sources are welcome!

[allyssonallan]

Congrats for the success! I am looking forward for new alternatives to test it.

[Yucheng-Jiang]

Update: We're gradually rolling out emails to users to opt-in email notification!

[allyssonallan]

Superb @Yucheng-Jiang I tested today! It is really cool!

[HolgerGelhausen]

Thank you very much, I have installed the software on my Mac. The research function works wonderfully. But the Writer gives errors, is the message to be understood as meaning that local writing is no longer possible?

[Yucheng-Jiang]

@HolgerGelhausen would you mind attach your error messages, steps to reproduce?

[HolgerGelhausen]

yes of course:

python -m scripts.run_writing --input-source console --engine gpt-4 --do-polish-article --remove-duplicate
Topic: ai_storytelling
sentence_transformers.SentenceTransformer : INFO : Load pretrained SentenceTransformer: paraphrase-MiniLM-L6-v2
/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
sentence_transformers.SentenceTransformer : INFO : Use pytorch device: cpu
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 94, in
main(parser.parse_args())
File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 54, in main
runner.run(topic=topic,
File "/Users/holgergelhausen/storm-main/src/engine.py", line 414, in run
article = self._generate_article(topic, outline, url_to_info, callback_handler)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/engine.py", line 26, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/engine.py", line 301, in _generate_article
section_result, section_refs = future.result()
^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/engine.py", line 276, in gen_section
url_to_snippets = search_collected_info.search(search_qs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/modules/write_page.py", line 220, in search
sim = cosine_similarity([encoded_query], self.encoded_snippets)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/utils/_param_validation.py", line 214, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/metrics/pairwise.py", line 1578, in cosine_similarity
X, Y = check_pairwise_arrays(X, Y)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/metrics/pairwise.py", line 173, in check_pairwise_arrays
Y = check_array(
^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/utils/validation.py", line 938, in check_array
raise ValueError(
ValueError: Expected 2D array, got 1D array instead:
array=[].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

[Yucheng-Jiang]

@HolgerGelhausen the error starts from sim = cosine_similarity([encoded_query], self.encoded_snippets)[0], which suggests that either encoded_query or encoded_snippets is empty. I highly suspect at early stages of pre-writing (research functions), it fails to retrieve any information snippets. To check if this is the case, could you check if observe a bunch of root : ERROR : Error occurs when searching query xxx: 'hits'? If this is the case, it's probably due to You.com api rate limit exceeds.

[HolgerGelhausen]

Thanks for help!!

I did not have the full overview of Storm. I will check the parameters later, this is the folder of the search.

Thanks for helping!!

[Yucheng-Jiang]

@HolgerGelhausen of course! the file raw_search_results.json contains the "research function" results. It should be empty if search engine api (you.com api) could successfully retrieve information. Feel free to follow up in this thread or open new issues if you run into any problem.

[HolgerGelhausen]

Thank you very much, yes it was a typo in the you apikey. I got an article now (after running the pre-stage again), but at the end i got this error message, something to look at? Thanks for help! Great Product!
storm_gen_article.txt

engine : INFO : _generate_article executed in 42.2798 seconds
Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
httplib_response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 1390, in getresponse
response.begin()
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 325, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 294, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/packages/six.py", line 769, in reraise
raise value.with_traceback(tb)
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request
httplib_response = conn.getresponse()
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 1390, in getresponse
response.begin()
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 325, in begin
version, status, reason = self._read_status()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 294, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_requestor.py", line 606, in request_raw
result = _thread_context.session.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 94, in
main(parser.parse_args())
File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 54, in main
runner.run(topic=topic,
File "/Users/holgergelhausen/storm-main/src/engine.py", line 419, in run
polished_article = self._polish_article(topic, article, remove_duplicate, callback_handler)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/engine.py", line 26, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/engine.py", line 343, in _polish_article
polish_result = polish_page(topic=topic, draft_page=article, polish_whole_page=remove_duplicate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dspy/primitives/program.py", line 29, in call
return self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/modules/write_page.py", line 295, in forward
page = self.polish_page(draft_page=draft_page).page
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dspy/predict/predict.py", line 60, in call
return self.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dspy/predict/predict.py", line 87, in forward
x, C = dsp.generate(signature, **config)(x, stage=self.stage)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/primitives/predict.py", line 78, in do_generate
completions: list[dict[str, Any]] = generator(prompt, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/holgergelhausen/storm-main/src/modules/utils.py", line 73, in call
response = self.request(prompt, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry
ret = target(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 136, in request
return self.basic_request(prompt, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 109, in basic_request
response = chat_request(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 247, in chat_request
return _cached_gpt3_turbo_request_v2_wrapped(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/cache_utils.py", line 17, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 221, in _cached_gpt3_turbo_request_v2_wrapped
return _cached_gpt3_turbo_request_v2(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/joblib/memory.py", line 655, in call
return self._cached_call(args, kwargs)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/joblib/memory.py", line 598, in _cached_call
out, metadata = self.call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/joblib/memory.py", line 856, in call
output = self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 216, in _cached_gpt3_turbo_request_v2
return cast(OpenAIObject, openai.ChatCompletion.create(**kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 25, in create
return super().create(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 155, in create
response, _, api_key = requestor.request(
^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_requestor.py", line 289, in request
result = self.request_raw(
^^^^^^^^^^^^^^^^^
File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_requestor.py", line 619, in request_raw
raise error.APIConnectionError(
openai.error.APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[HolgerGelhausen]

Thanks for helping @Yucheng-Jiang , i wrote an Blog Post in German about Storm with an example about AI Storytelling!! A Game Changer in Content Creation!

https://www.ki-insights.com/110-der-perfekte-artikel-das-storm-projekt-von-stanford/

[Yucheng-Jiang]

@HolgerGelhausen Here's a thread discussing this error: https://community.openai.com/t/api-aborts-my-connection-without-a-reason-anything-i-can-do/104256/35. It should be something to do with connection with openAI api endpoints.

There's one people in above thread suggesting

Turning off my VPN and it works.

[shaoyijia]

Update: We're gradually rolling out emails to users to opt-in email notification!

Latest Update: Unfortunately, the very high demand exceeds our lab budget, so we have to disable the "Create New Article" function again. 4.5k+ people have tried out this web preview and many left very helpful feedback. Thank you so much!!

We are figuring out the possibility to keep hosting the web preview as many people find it useful. In the meanwhile, we are working on refactoring our codebase and planning the development roadmap with the vision of generalizing STORM into an easy-to-use and easy-to-extend knowledge curation engine. Stay tuned!

[HolgerGelhausen]

@shaoyijia The high demand is a great compliment for your demand, the fact that the budget is exceeded is therefore also understandable. Great work and thank you for taking the work to share it here! Thank you for answering and helping!!