Case Study: VDI/VDE
With Elicit, we can produce a very high quality review for clients in a couple of months. That’s a valuable selling point.” Veit Wasserfuhr, Consultant, VDI/VDE-IT
VDI/VDE Innovation + Technik GmbH is a German consulting firm that advises governments and the private sector on innovation and technology. In 2023, the company employed more than 1,000 subject-matter experts and managed ~3 billion euros for clients.
VDI/VDE-IT uses Elicit to evaluate 11x more papers per project, enabling them to offer clients more ambitious and rigorous systematic reviews.
Before Elicit: Constrained by resources
VDI/VDE-IT had limited capacity to conduct and market in-house systematic reviews.
Veit Wasserfuhr, Consultant, said, “In a normal industry systematic literature review, it’s typically impossible to closely analyze more than 50-100 papers manually. It’s painstaking and inefficient. There’s not enough time to go deep on each paper. You have to scan papers quickly and report with limited information.”
They dealt with these constraints in part by outsourcing systematic reviews, but this made it difficult for them to control quality. Wasserfuhr adds, “When we outsource systematic reviews, we pay the company and get a report back. We don’t get to be involved in drafting it.”
Choosing Elicit: A formal evaluation
Wasserfuhr took the initiative to try and solve these longstanding issues with a tool he trusted: “I started using Elicit 4 years ago before the ChatGPT hype and was already really impressed. I followed the newsletters and saw continuous improvement on the fundamentals.”
To validate this first impression, VDI/VDE-IT rigorously compared Elicit’s extraction to their own manual extraction of 50 papers. The results were, in Wasserfuhr’s words, "very impressive": Elicit was 99.4% accurate. “In some extractions, it provided more comprehensive answers than manual reviewers.”
Today: Using Elicit to evaluate 11x more papers per systematic literature review
The firm recently used Elicit to expand the scope of a systematic review on educational interventions from 50 papers to 550 papers. Commenting on this, Wasserfuhr said, “With Elicit, we can produce a very high quality review for clients in a couple of months. That’s a valuable selling point.”
Dr. Ernst Andreas Hartmann, Head of VDI/VDE-IT’s Education consulting division, added “What I really liked about Elicit is that you can check the references behind a result within the context of the original paper. Elicit’s claim verification is really, really good and gives me confidence in using it.”
Tomorrow: Scaling Elicit across the company
In March, Hartmann and Wasserfuhr plan to showcase Elicit to other departments.
“There is quite a bit of interest.” Wasserfuhr said “Our researchers want to use Elicit not only for systematic literature reviews, but also for scoping reviews, rapid reviews, and trend analysis.” He detailed a project where VDI/VDE-IT plans to analyze trends in the abstracts of more than 50,000 papers.
When discussing his experiences with competitor tools, Wasserfuhr summed it up with: “For data extraction in rigorous systematic literature reviews, I trust Elicit.”