start policy updates on ai tool usage#1006
Conversation
|
Any ETA? We could aim for a small release that'd have this and the other changes since the last release. I know this is all still very WIP. |
Thanks for the prod - I think this should be merged as quickly as possible. I also think it's okay for an initial first draft of our policies. I hope and expect we'll edit and revise as things proceed, but am happy for you to tweak, approve, and merge when you're ready. Thanks! |
|
I'll generate the translations ASAP. |
maelle
force-pushed
the
ai-updates
branch
4 times, most recently
from
c191b55 to
7220acd
Compare
maelle
requested review from
beatrizmilz and
yabellini
and removed request for
a team
|
@maelle I also just added another checkbox to templates/review.md which will also need translating. |
|
@mpadge I added the missing translations. In such cases I use copy-paste (DeepL interface or https://docs.ropensci.org/babeldown/reference/deepl_translate_clipboard.html), I can only use |
yabellini
left a comment
yabellini
left a comment
There was a problem hiding this comment.
Some Spanish suggestions.
Some comments on the clarity of the instructions.
There is text that is repeated twice; it looks like we should delete one.
| Como se indica en [nuestras políticas generales](#policies-ai) exigimos a todos los autores que envíen propuestas que describan cómo se han utilizado dichas herramientas, y que incluyan enlaces a cualquier aspecto relevante de los repositorios. | ||
| Una regla muy general es que cuanto mayor sea el uso de herramientas de IA generativa en un envío a rOpenSci, más documentación esperaremos y más esperaremos que se hayan utilizado dichas herramientas. *sistemáticamente*. | ||
| El uso sistemático de herramientas incluye cualquier enfoque que detalle de forma transparente y progresiva la contribución de las herramientas de IA generativa a la producción de software. | ||
| Los ejemplos de herramientas sistemáticas de IA generativa van desde [el "spec-kit" de GitHub para el desarrollo basado en especificaciones](https://github.com/github/spec-kit) a [nuestra propia herramienta experimental para documentar las decisiones de diseño de software](https://github.com/ropensci-review-tools/designlens). |
There was a problem hiding this comment.
I have some problems understanding what I'm supposed to do if I use AI in my package to meet the requirements outlined here. I need to check the examples linked in more deeply. I'm wondering if we can link to good examples from a package under review or other packages that have the information as we would like to see.
| El uso de herramientas de IA generativa es aceptable tanto para los autores como para los revisores, pero los revisores no pueden utilizarlas para fundamentar directamente sus decisiones o recomendaciones. | ||
| Por lo general, los revisores deben solicitar la aprobación de los autores y editores antes de utilizar herramientas de IA generativa, y hacerlo públicamente en el hilo de la revisión. | ||
| Al igual que esperamos de los autores, exigimos a los revisores que revelen cualquier uso de herramientas de IA generativa con el mayor detalle posible. | ||
| Algunos ejemplos de uso aceptable son la generación de una visión general de la funcionalidad del paquete, o la iteración a través de un historial Git para proporcionar una visión general de las decisiones de desarrollo. |
There was a problem hiding this comment.
Can we link to some good examples? Or how can you actually do that?
| Examples of acceptable usage include generating an overview of package functionality, or iterating through a Git history to provide an overview of development decisions. | ||
| We ask reviewers to be as transparent as possible. | ||
|
|
||
| ### Use of generative AI tools |
There was a problem hiding this comment.
This looks like it is repeated. It is the same text two times.
I removed it in the Spanish version. I advice to remove in the Portuguese one, I also suggest to remove from here.
Co-authored-by: Yanina Bellini Saibene <yabellini@gmail.com>
Co-authored-by: Yanina Bellini Saibene <yabellini@gmail.com>
Co-authored-by: Yanina Bellini Saibene <yabellini@gmail.com>
Co-authored-by: Yanina Bellini Saibene <yabellini@gmail.com>
Co-authored-by: Yanina Bellini Saibene <yabellini@gmail.com>
Co-authored-by: Yanina Bellini Saibene <yabellini@gmail.com>
|
@mpadge @yabellini has great comments on the content. Could you please work on those, and then request a new review from us both? I've removed the PT translation request for now. |
|
@mpadge would it be worth starting a new PR with only the English changes that @yabellini and i could review, and then adding the translations? |
| This kind of information can also be automatically extracted from sufficiently detailed Git logs (and our [designlans tool](https://github.com/ropensci-review-tools/designlens) includes functionality to do just that). | ||
| The Git log remains the ultimate source of insight into design history, and a minimal requirement for all submissions remains | ||
| - The use of generative AI tools is acceptable in packages submitted for peer review, as described in [our initial blog post](https://ropensci.org/blog/2026/02/26/ropensci-ai-policy/). | ||
| - As stated in [our general policies](#policies-ai), we require all submitting authors to describe how such tools may have been used, and to include links to any relevant aspects of repositories. |
There was a problem hiding this comment.
details on such links? Example of a GitHub permalink?
There was a problem hiding this comment.
No great ones yet - definitely planned, but nothing to date is in a state that would be a good reference there.
There was a problem hiding this comment.
I wasn't clear enough: I think this should explain the format you expect for such links, probably permalinks.
|
|
||
| #### How to use generative AI tools in preparing software for review | ||
|
|
||
| Software review must be able to focus on design decisions and design history. |
There was a problem hiding this comment.
Why history? What does it mean in practice for a reviewer? I'm not playing devil's advocate, I just think that as a reviewer I wonder.
How about creating an example fake submission to exemplify what you have in mind?
There was a problem hiding this comment.
I like the idea of having a toy example. I read this, and to be honest, I don't know what to do as an author. Many people who send their packages to our peer review are not experienced developers or even have formal training in developing software.
| Generative AI tools should be used in ways that best enable that. | ||
|
|
||
| - We encourage the use of tools which produce systematic records of design decisions and histories, such as [GitHub's "spec-kit"](https://github.com/github/spec-kit) or [our own experimental 'designlens' tool](https://github.com/ropensci-review-tools/designlens). | ||
| - Both of these create an additional `specs/` directory intended to be committed within a repository, and that records individual development phases, generally under sequentially-numbered sub-directories, with each containing several documents like "plan.md", "tasks.md", and "report.md". |
There was a problem hiding this comment.
aren't they _very_verbose? The one I've tried out for a hackathon once is https://github.com/Fission-AI/OpenSpec/
There was a problem hiding this comment.
Yeah, they're verbose, but the outputs really are very useful to understanding human decision-making processes behind software. In my opinion, this kind of verbosity is currently the best way to ensure something human is retained amidst the machine-generated noise. And it all adds only one additional sub-dir within a repo, which ain't that much.
There was a problem hiding this comment.
but part of the verbosity comes from the IA not the humans describing their thoughts?
There was a problem hiding this comment.
Trying to understand the process here: I developed a package, at some point I use AI, then I decide to send the package to peer-review, and then I read this policy and learn about the existence of designlens and the GitHub tool. Can I use it at this stage? Or do these tools need to be used from the start of the development to capture the design decisions and history?
|
|
||
| - We encourage the use of tools which produce systematic records of design decisions and histories, such as [GitHub's "spec-kit"](https://github.com/github/spec-kit) or [our own experimental 'designlens' tool](https://github.com/ropensci-review-tools/designlens). | ||
| - Both of these create an additional `specs/` directory intended to be committed within a repository, and that records individual development phases, generally under sequentially-numbered sub-directories, with each containing several documents like "plan.md", "tasks.md", and "report.md". | ||
| - A `specs/` folder can also be produced and maintained by hand, or by directly instructing a generative AI tool, but however produced it should contain a comprehensive history of software design, and provide an informative basis for review. |
There was a problem hiding this comment.
I really think we need an example package of some sort.
There was a problem hiding this comment.
Agreed, but see comment above - I prefer to wait until we have some good submissions to link to. I expect we'll be iterating on these phrases quite frequently, and really, strongly hope to be able to have good ones to link to. They're just not there yet.
|
|
||
| - [ ] Se utilizaron herramientas de IA generativa para realizar esta revisión | ||
|
|
||
| Solo si marca esa casilla: explique la contribución de dichas herramientas a su revisión. |
There was a problem hiding this comment.
| Solo si marca esa casilla: explique la contribución de dichas herramientas a su revisión. | |
| Solo si marca esta casilla: explique la contribución de dichas herramientas a su revisión. |
| ### Uso de herramientas de IA generativa | ||
|
|
||
| El uso de herramientas de IA generativa es aceptable en los paquetes enviados para revisión por pares, como se describe en [nuestro artículo en el blog](https://ropensci.org/blog/2026/02/26/ropensci-ai-policy/). | ||
| Como se indica en [nuestras políticas generales](#policies-ai) exigimos a todas las personas autoras que envíen propuestas que describan cómo se han utilizado dichas herramientas, y que incluyan enlaces a cualquier aspecto relevante de los repositorios. |
There was a problem hiding this comment.
| Como se indica en [nuestras políticas generales](#policies-ai) exigimos a todas las personas autoras que envíen propuestas que describan cómo se han utilizado dichas herramientas, y que incluyan enlaces a cualquier aspecto relevante de los repositorios. | |
| Como se indica en [nuestras políticas generales](#policies-ai) pedimos a todas las personas que envían sus paquetes a nuestra revisión que describan cómo se han utilizado dichas herramientas y que incluyan enlaces a cualquier aspecto relevante de los repositorios. |
There was a problem hiding this comment.
Can we add examples of what a relevant aspect is ? As a potential author sending a package I don't know what a relevant aspect is...
There was a problem hiding this comment.
@yabellini Unfortunately, not yet. We don't yet have any good examples to link to. We just need to get these updates out first, then hope to have actual examples in as quickly as possible after then.
|
|
||
| El uso de herramientas de IA generativa es aceptable en los paquetes enviados para revisión por pares, como se describe en [nuestro artículo en el blog](https://ropensci.org/blog/2026/02/26/ropensci-ai-policy/). | ||
| Como se indica en [nuestras políticas generales](#policies-ai) exigimos a todas las personas autoras que envíen propuestas que describan cómo se han utilizado dichas herramientas, y que incluyan enlaces a cualquier aspecto relevante de los repositorios. | ||
| Una regla general es que, cuanto más se utilicen herramientas de IA generativa en en un paquete enviado a rOpenSci, más documentación esperaremos recibir. Además esperaremos que dichas herramientas se hayan utilizado *sistemáticamente*. |
There was a problem hiding this comment.
| Una regla general es que, cuanto más se utilicen herramientas de IA generativa en en un paquete enviado a rOpenSci, más documentación esperaremos recibir. Además esperaremos que dichas herramientas se hayan utilizado *sistemáticamente*. | |
| Una regla general es que, cuanto más se utilicen herramientas de IA generativa en un paquete enviado a rOpenSci, más documentación esperaremos recibir. Además, esperaremos que dichas herramientas se hayan utilizado *sistemáticamente*. |
There was a problem hiding this comment.
It said more documentation we wait to get, this documentation is about the use of AI? It can be confused with package documentation, at least in the Spanish version, and also what kind of documentation do we expect to receive? We really need examples here.
yabellini
left a comment
yabellini
left a comment
There was a problem hiding this comment.
I still haven't reviewed the Spanish version because I have more comments and questions. I always like our documentation because it is clear and helpful, and I know we are in unknown territory here, so I understand we don't have all the answers.
|
|
||
| #### How to use generative AI tools in preparing software for review | ||
|
|
||
| Software review must be able to focus on design decisions and design history. |
There was a problem hiding this comment.
I like the idea of having a toy example. I read this, and to be honest, I don't know what to do as an author. Many people who send their packages to our peer review are not experienced developers or even have formal training in developing software.
| Generative AI tools should be used in ways that best enable that. | ||
|
|
||
| - We encourage the use of tools which produce systematic records of design decisions and histories, such as [GitHub's "spec-kit"](https://github.com/github/spec-kit) or [our own experimental 'designlens' tool](https://github.com/ropensci-review-tools/designlens). | ||
| - Both of these create an additional `specs/` directory intended to be committed within a repository, and that records individual development phases, generally under sequentially-numbered sub-directories, with each containing several documents like "plan.md", "tasks.md", and "report.md". |
There was a problem hiding this comment.
Trying to understand the process here: I developed a package, at some point I use AI, then I decide to send the package to peer-review, and then I read this policy and learn about the existence of designlens and the GitHub tool. Can I use it at this stage? Or do these tools need to be used from the start of the development to capture the design decisions and history?
| - We encourage the use of tools which produce systematic records of design decisions and histories, such as [GitHub's "spec-kit"](https://github.com/github/spec-kit) or [our own experimental 'designlens' tool](https://github.com/ropensci-review-tools/designlens). | ||
| - Both of these create an additional `specs/` directory intended to be committed within a repository, and that records individual development phases, generally under sequentially-numbered sub-directories, with each containing several documents like "plan.md", "tasks.md", and "report.md". | ||
| - A `specs/` folder can also be produced and maintained by hand, or by directly instructing a generative AI tool, but however produced it should contain a comprehensive history of software design, and provide an informative basis for review. | ||
| - For alternative approaches, consider JOSS's requirement for statements of [_Scope and significance_](https://joss.readthedocs.io/en/latest/submitting.html#scope-and-significance), and see [recently accepted papers](https://joss.theoj.org/papers/published) for examples of such statements. |
There was a problem hiding this comment.
Nice that we have some examples!
3 participants
Checklist for dev guide maintainers, do not delete 😸