This is the information page for the 1 1/2 hour Quantum Computing Benchmarking Working Group (QCB-WG) P7131 meeting. The meeting is Tuesday, August 23, 2022 10 AM U. S. Eastern Daylight time via Zoom. The Zoom room will operate for a total of two hours if there is a need for informal discussion.
Main topic is talk:
Title: Metriq: a platform for community-driven quantum benchmarks
Coauthors: Daniel Strano (Unitary Fund) and Vincent Russo (Unitary Fund)
With the growth of quantum computing in academia and industry, it can be hard to keep track of the many new developments. The community lacks an open repository of “state-of-the-art” benchmark results, and it is difficult to know how to appropriately compare “like-to-like,” across the breadth of published quantum benchmark results. Metriq is designed to address these challenges.
Metriq is an open source web application for reporting, presenting, analyzing, and tracking benchmarks in the domain of quantum technologies. “Submissions” to the app can be any URL that constitutes a source for quantum benchmarks, presenting “results” at the intersection of a “method” and a “task.” “Results” are reported by the community of users at the intersection of quantum technology “methods” performing desired “tasks,” or workloads of interest. Metriq publicly presents comparisons of “results” from “methods” across any given “task.”
We will briefly explain the motivating challenges of quantum benchmarks that Metriq is meant to address, followed by a “tour” of the live web application at https://metriq.info, explaining its taxonomy and workflows.
The plan is to have a P7131 business meeting in the first 15 minutes, including determination of quorum, etc. People not part of P7131 can join 15 minutes into the meeting when Dan Strano will start the presentation. The formal presentation is expected to be 15-20 minutes followed by discussion. The meeting will officially end after 1 1/2 hours, but discussion can continue via Zoom until the 2 hour point.
This meeting is open to all that are interested. All you need to attend is the Zoom connection information that appears at the bottom of this page. We will take attendance in accordance with IEEE-SA’s policies.
Agenda:
10:00 AM EDT: Call to order
10:00 AM EDT: Establish voting membership
10:02 AM EDT: Call for (a) approval of minutes from three past meetings (see below) and (b) changes to this draft agenda, followed by approval.
10:05 AM EDT: IEEE-SA policies
10:15 AM EDT: Presentation on Metriq by Dan Strano and/or Vincent Russo.
10:35 AM EDT: Question and answer / discussion
11:00 AM EDT: Meeting officially adjourns
11:00 AM EDT: Discussion may continue if needed
noon EDT: Must end by this time
This webpage https://sagroups.ieee.org/7131/qcb-wg-aug22/ is expected to be maintained, so check back
—————————–
Erik DeBenedictis is inviting you to a scheduled Zoom meeting.
Topic: IEEE QCB-WG P7131 meeting (topic: Metriq)
Time: Aug 23, 2022 10:00 AM Eastern Time (US and Canada)
Join Zoom Meeting
https://us02web.zoom.us/j/88158717202?pwd=ai9mNFVaMjZEcDZickM0ZFIvV1h2dz09
Meeting ID: 881 5871 7202
Passcode: 059609
One tap mobile
+16699009128,,88158717202#,,,,*059609# US (San Jose)
+17193594580,,88158717202#,,,,*059609# US
Dial by your location
+1 669 900 9128 US (San Jose)
+1 719 359 4580 US
+1 253 215 8782 US (Tacoma)
+1 346 248 7799 US (Houston)
+1 669 444 9171 US
+1 564 217 2000 US
+1 646 558 8656 US (New York)
+1 646 931 3860 US
+1 301 715 8592 US (Washington DC)
+1 309 205 3325 US
+1 312 626 6799 US (Chicago)
+1 386 347 5053 US
Meeting ID: 881 5871 7202
Passcode: 059609
Find your local number: https://us02web.zoom.us/u/kd1HYFeJ0D
——————————
Minutes February 28, 2022 meeting
(a) P7131/QCB-WG
10:00 AM EST (b) Meeting via Zoom called to order 10 AM EST (c) Chair Erik DeBenedictis, presiding and writing (these) minutes as secretary.
10:05 AM EST Introductions (d) and establishment of voting membership. The following are derived from contemporaneous notes and may contain spelling errors. The contemporaneous notes contain more data. The information in the chair’s contemporaneous notes has been merged with Jonathan Goldberg’s notes to create the QCB-WG and QCB-WG-MBRS LISTSERV group lists.
Mike Pendergrast. Vote.
Alexandru Paler. Vote.
Jonathan Goldberg. IEEE SA Staff.
Santanu Basu. Corning. Vote.
Bruce Kraemer. Vote.
Dick Barr. Vote.
John Spargo. Vote.
Ellen Winsor. Vote.
Alvin Chin.
Andrew Ochoa. No vote.
Baw Chng. Bawman LLC, reps self. Vote.
Brice Achir, Cisco. Voting.
Charlie Baldwin, Quantumuum. No vote.
Claudio Lima, BC Vote
Cathy Mcgeoch, D-Wave, vote.
Dan Strano. Unitary fund. Vote.
Dan Gottesman, Keysight/representing self.
Dipankar. Vote.
Esam El-Araby. Vote.
Florian Price. No vote.
Ivan Rungger. UK National Measurement. Vote.
Jennifer Harding. Q. Brilliance. Voting.
John Devaney. UK National phy laboratory. Vote.
John Greaves. Vote yes.
K.-C. Chen. Univ Southbury. Vote.
Kevin Young. Sandia. No vote.
Masahiro Horbe. Vote.
Maurizio Ferrari Dacrema. Post doc Polytech Milan. No vote.
Mingchange Wang. Hardware from Dell tech. Vote.
Phillip Ross. No vote.
Raj Chanian. UK cap ni? Vote.
Reena Dayal. IEEE QI also represent Q. Ecosystems Tech council of India. No vote.
Rajesh. Mo… Management consulting company. No vote.
Sara Biyabani. From Si. Valley? Vote.
Scott Glancy. NIST. Vote yes.
Scott Holmes. Works for BAH. Vote.
SD. Das. India. No vote.
Sebastian. Rockwell. No vote.
Seth Merkel. Physicist IBM. Vote yes.
Simon Caton. Asst Prof CS U. College Dubin. Vote.
Sri Chan… IEEE staff. No vote.
Steve Bush. GE Research. No vote.
Surat Hazra “gam genini”? No vote.
Terrill Frantz. Harrisburg U. Vote.
Victor Rodriguez-Toro. IBM.
Victoria. Certified auditor. Vote.
Vladimir Getov. CS Prof U. Westminister London. Vote.
Yan-Ping Lu. Monash Univ. Vote.
Angie Quarry. Vote.
Paolo Gargini. Yes vote.
Sara Kaiser. Unitary fund. Voting.
Fujitsu Hiroki, Itoh. No vote.
10:00 AM EST (e) N/A, this was a kickoff meeting.
11:05 AM EST Call for changes to agenda: none
11:05 AM EST(f) Approval of agenda: Motion McGeoch Second Winsor, pass unanimous consent
11:08 AM EST Jonathan Goldberg read (g) patent, (h) copyright, participation slide decks
(i) no previous meeting.
DeBenedictis, Vladimir Getov, Angie Qarry, Dick Barr, Cathy McGeoch
(l) There was no executive session.
11:10 AM EST. Erik DeBeneductis discussed slide deck (j)
No action items (k), no executive session (l)
Motion to end meeting: Motion Baw Second Barr pass unanimous consent
11:20 AM EDT End of informal discussion
(n) The next meeting will be scheduled by email, Doodle, etc. (but there was no discussion here).
Minutes May 10, 2022
(a) P7131/QCB-WG
9:00 AM EDT (b) Meeting via Zoom called to order 9 AM EDT (c) Chair Erik DeBenedictis, presiding and writing (these) minutes as secretary.
9:00 AM EDT (d) Establish voting membership (e) It was initially assessed there were not enough members present, but a more careful tally after then meeting revealed that late arrivals resulted in the exact minimum quorum (see attendance list). No votes were needed.
Attendence list:
Abhishek Agarwal, National Physical Laboratory, UK
Jonathan J. Attia, Feynman Foundation
Richard (Dick) Barr, Southern Methodist University
Simon Caton, University College Dublin
Raj Chanian, Capgemini
Baw Chng, BAWMAN LLC
Shamik Das, The MITRE Corporation
Shouvik Datta, IISER-Pune
Erik DeBenedictis, self
John Devaney, National Physical Laboratory UK
Esam El-Araby, University of Kansas (KU)
Terrill Frantz, Harrisburg University
Joshua Gay, IEEE
Vladimir Getov, University of Westminster
Victoria (Vicky) Hailey, vhg
Jennifer Harding, Quantum Brilliance
Hiroshi Kawamura, Fujitsu ltd.
Bruce Kraemer
Catherine McGeoch, dwavesys.com
Alexandru Paler
Angie Qarry, Disruptive Innovations
John Spargo, Northrop Grumman
Wen Wei, Fujitsu Limited
9:05 AM EDT Call for changes to agenda: none
9:05 AM EDT(f) Approval of agenda: Motion McGeoch Second Barr pass unanimous consent
9:08 AM EDT Read (g) patent, (h) copyright, participation slide decks
(i) Oops, missed approval of minutes of previous meeting.
9:15 AM EDT (j) Introduce Silona Bonewald, who presented the slide deck below:
SA-OPEN-default-slide-deckDownload
“Assignments” (k):
Cathy to send template for benchmark to Erik; Erik to forward to QCB-WG; WG email reflector to be used to assemble first cut at a list of benchmarks
Initial group for assembling a list of benchmarks and possible some speakers at meeting: Erik DeBenedictis, Vladimir Getov, Angie Qarry, Dick Barr, Cathy McGeoch
(l) There was no executive session.
9:59 AM EDT (m) Motion to end meeting: Motion Barr Second McGeoch pass unanimous consent
10:20 AM EDT End of informal discussion
(n) The next meeting will be scheduled by email, Doodle, etc. (but there was no discussion here).
Minutes June 28, 2022
(a) P7131/QCB-WG
10:00 AM EDT (b) Meeting via Zoom called to order 10 AM EDT (c) Chair Erik DeBenedictis, presiding and writing (these) minutes as secretary.
10:00 AM EDT (d) Establish voting membership (e) It was initially assessed there were not enough members present, but a more careful tally after the meeting revealed that 19 voting members attended, which met the minimum quorum (see attendance list). However, there were not enough members present at the beginning of the meeting to approve past minutes.
Attendance list:
Erik DeBenedictis
Bruce Kraemer
Shouvik Datta
Esam El-Araby
Jonathan Goldberg
Richard Barr
Simon Caton
Terrill Frantz
Vladimir Getov
Anne Sophie Barbe
Victoria (Vicky) Hailey
Angie Qarry
Ivan Rungger
Tom Lubinski
Jennifer Harding
brice achkir
santanu basu
Panu Vesanen
Masahiro Horibe
Catherine McGeoch
Claudio Lima
Raj Chanian
Baw Chng
Kazutomo Hasegawa
Skamik Das
Yan-Ping Lu
John Qu
8:05 AM EDT Call for changes to agenda: none
8:05 AM EDT(f) Approval of agenda: [forgot about this step]
9:08 AM EDT Read (g) patent, (h) copyright, participation slide decks
(i) No approval of minutes of previous meeting to do lack or quorum as of this time.
9:15 AM EDT (j) Introduce Tom Lubinski, who presented the slide deck below:
[[*** TOM’s SLIDE DECK ***]]
Tom: Discussion of IEEE 7131. Vetting QED-C approach
Erik: Discussion of trial-use standard when real standard premature
Esam says state preparation and measurement is missing from existing benchmarks and should be added. Tom Lubinski says this is a great idea.
Angie: IEEE and QED-C could get voice of industry involved in what benchmarks are needed. How to get use more involved.
Vicky: Trial use standards are tech reports for ISO; this group similar to ISO AI group; when IEEE fast tracks to ISO, ISO can change.
Vladimir: Liked presentation. Should call what’s going on now (execution time measurements) a “methodology.” Tom: QED-C focus has been comparing different platforms.
9:04: (m) Motion to adjourn Cathy, seconded Dick, passed unanimously (however, meeting continues informally for another 30 mins).
Angie: Corporations interested in runtime. For example quantum 1 minute, classical 3 minutes.
Tom on action items for QED-C & P7131: (1) QED-C has an open source repository for those who can run benchmarks (2) Joint meeting between IEEE + QED-C contributors
Simon: Quantum machine learning used data sets (will provide link to a paper) Conclusions: Shallow circuits favored for reasons beyond just low noise.
Cathy: Part of a group doing an “industry manifesto.” What is sought from “makers” is metrics in $ (money) not runtime.
Meeting ends.
“Assignments” (k): Discussion of a meeting with more people, P7131 access to QED-C repository, and IEEE could vet QED-C benchmarks.
(l) There was no executive session.
9:40 AM EDT End of informal discussion
(n) The next meeting will be scheduled by email, Doodle, etc. (but there was no discussion here).
Minutes August 23, 2022
(a) P7131/QCB-WG
10:00 AM EST (b) Meeting via Zoom called to order 10 AM EST (c) Chair Erik DeBenedictis, presiding and writing (these) minutes as secretary.
10:05 (d) establishment of voting membership. The chair called names in groups of three, asking for proclamations of “I’m here,” and writing results on a piece of paper. Chair also posted a URL for a Google form in the Zoom chat box. The chair reconciled the verbal proclamations with the results in the Google form after the meeting and produced the table below.
| Name, affiliation | See text |
| Richard (Dick) Barr, Southern Methodist University | 4 |
| Simon Caton. Asst Prof CS U. College Dubin. | 4 |
| Raj Chanian. UK cap gemini | 4 |
| Kwang Cheng | |
| Baw Chng. Bawman LLC, reps self. Vote. | 4 |
| Shouvik Datta, IISER-Pune | |
| Erik DeBenedictis | Chair |
| Terrill Frantz. Harrisburg U. | 4 |
| Kostas Georopolous | request 1 |
| Vladimir Getov. CS Prof U. Westminister London. | 4 |
| Luke Govia, IBM | |
| Kazutomo Hasegawa | |
| Nils Herrmann | request 1 |
| Hiroshi Kawamura, Fujitsu ltd. | |
| Bruce Kraemer | 4 |
| Seth Merkel. Physicist IBM | 2 |
| Kevin Obenland | request 1 |
| Alexandru Paler | 2 |
| John Qu | 2 |
| Vincent Russo | request 1 |
| Dan Strano. Unitary fund | 2 |
| Panu Vesanen, iqm | |
| Wen Wei, Fujitsu Limited |
The interpretation of the second column is as follows: The field is blank for attendees who have not asked to vote. If the field contains the number 1-4, this will be how many of the last four meetings the person will have attended AT THE NEXT MEETING. If the person requested voting privileges for the first time at this meeting, the word “request” appears in this field as well. The table is not a membership list because it does not include members in good standing the did not attend this meeting. Note: According to the chair’s records, 10 people attended the February 28 organizational meeting but no other meetings. Due to the requirement of attending 2 of the last 4 meetings to vote, these people will not count for quorum nor will they be able to vote at the 5th meeting (the next meeting). After excluding these 10 people and considering new members there will be 25 voting members at the time of the next meeting (including the chair), so quorum will be 13. The information in the table and this paragraph has not been approved. Please report errors.
10:05 AM EST (e) Quorum was not met, so approval of minutes will be deferred.
10:05 AM EST(f) Additions to and approval of agenda. Sorry, forgot this item.
10:08 AM EST Erik DeBenedictis read (g) patent, (h) copyright, participation slide decks.
(i) No quorum
(l) There was no executive session.
10:15 AM EDT (j) Introduce Dan Strano and Vincent Russo, who presented the slide deck below:
[[*** PUT Dan’s SLIDE DECK HERE WHEN WE HAVE IT ***]]
10:59 AM EST. Chair called for motion to end meeting: Motion Barr Second Getov pass unanimous consent (but nobody dropped off at this point).
There was substantial discussion related to how IEEE standards function with regard to materials developed by members versus imported from the outside (such as through Metriq). Basically, DeBenedictis, Goldberg, and Kraemer described how there were two phases to an IEEE standard: In the first phase, the working group (i. e. this group) comes up with a draft standard (i. e. a document but possibly including software) and in a second phase a ballot group comments and votes on whether to make the draft document a standard. Based on a question from Govia, IEEE must approve the ballot group as representative of the community. If IEEE does not approve the working group will have to change/expand the ballot group and propose again. The working group has some options on creating the draft document: (a) Members could write the draft from scratch. (b) A subset of the members could offer material that they had prepared in advance (such as quantum volume or computational qubits) as a starting points and the members could merge and tweak. There was discussion initiatiated by Getov on benchmarks representative of complete applications versus (using informal language here) support subroutines.
The action items (k) were not formal, but Strano and Russo are interested in further engagement for Metriq. However, there was an implicit action item that one or more of the 200+ benchmark activities in Metriq could become a starting point for a draft IEEE standard.
Discussion mostly included DeBenedictis, Goldberg, Kraemer, Govia, Strano, Getov.
No executive session (l)
11:20 AM EDT End of informal discussion
(n) The next meeting will be scheduled by email, Doodle, etc. (but there was no discussion here).