Support need items generation from marked RST in source code

[juiwenchen]

Addressing #43 by adding simplified RST directive parser and feature to remove leading sequences of marked RST text

Documentation

• Added a dedicated section (docs/source/components/rst_parser.rst) describing the new simplified RST parser, its purpose, and limitations.

RST block extraction and parsing:

• Added a new parser module using Lark to parse extracted RST blocks, extracting directive type, argument, options, and content.
• Updated MarkedRstConfig and related configuration files to support new fields: strip_leading_sequences, indented_spaces, and link_options for more stable RST extraction and parsing.

Integration and warning handling:

• Integrated the RST parser into the extraction workflow in analyse, including logic to handle parsing errors and report warnings.
• Extended the MarkedRst model to store parsed need data from RST blocks.

[MrChocolateMoose]

We are currently using this feature (by using this branch instead of 1.1.0). I can confirm that this is working for us.

Was finishing the implementation and getting this PR merged planned for 1.2?

[juiwenchen]

We are currently using this feature (by using this branch instead of 1.1.0). I can confirm that this is working for us.

Was finishing the implementation and getting this PR merged planned for 1.2?

Didn't expect you tested it already! Very happy to receive such info, and thanks for it. I am currently working on its warning handling to be more user-friendly. And yes, this feature is planned for 1.2.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.08163% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.55%. Comparing base (0209874) to head (555de94).

Files with missing lines	Patch %	Lines
tests/conftest.py	54.54%	20 Missing and 5 partials ⚠️
src/sphinx_codelinks/analyse/sn_rst_parser.py	92.20%	3 Missing and 3 partials ⚠️
src/sphinx_codelinks/analyse/analyse.py	88.63%	2 Missing and 3 partials ⚠️
...codelinks/sphinx_extension/directives/src_trace.py	72.72%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #44      +/-   ##
==========================================
- Coverage   89.75%   89.55%   -0.21%     
==========================================
  Files          29       31       +2     
  Lines        2548     2776     +228     
  Branches      297      335      +38     
==========================================
+ Hits         2287     2486     +199     
- Misses        165      184      +19     
- Partials       96      106      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds support for generating Sphinx-Needs items from marked RST directives embedded in source code comments. It introduces a simplified RST directive parser using the Lark parsing library to extract and parse RST blocks, making it possible to define needs using full RST directive syntax within code comments.

Key Changes

• RST Parser Implementation: Added a new Lark-based parser (sn_rst_parser.py) to parse RST directive syntax from extracted comment blocks, extracting directive types, titles, options, and content
• Configuration Extensions: Extended MarkedRstConfig with three new fields (strip_leading_sequences, indented_spaces, and link_options) to handle various comment styles and RST parsing requirements
• Integration and Warning System: Integrated the parser into the analysis workflow with comprehensive error handling and warnings for invalid RST directives

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
src/sphinx_codelinks/analyse/sn_rst_parser.py	New simplified RST parser module using Lark with grammar definitions, preprocessing, and transformation logic
src/sphinx_codelinks/analyse/analyse.py	Integration of RST parser into extraction workflow with warning handling and link option processing
src/sphinx_codelinks/analyse/projects.py	Extended warning collection to include RST parsing warnings from all projects
src/sphinx_codelinks/analyse/models.py	Extended MarkedRst model to include parsed need data
src/sphinx_codelinks/analyse/utils.py	Updated RST extraction logic with improved row offset calculation
src/sphinx_codelinks/config.py	Added new configuration fields for RST parsing behavior
src/sphinx_codelinks/sphinx_extension/directives/src_trace.py	Updated directive to handle both one-line needs and parsed RST blocks
src/sphinx_codelinks/cmd.py	Added call to update_warnings in analyse command
tests/test_rst_parser.py	Comprehensive test suite for RST parser with positive and negative test cases
tests/test_analyse.py	Added integration tests for RST extraction and parsing
tests/fixture_files/analyse_rst.yml	Test fixtures for various RST marker scenarios
tests/conftest.py	Added test infrastructure for fixture file parametrization
tests/data/marked_rst/dummy_1.cpp	Fixed spacing in inline RST marker
tests/data/dcdc/charge/demo_2.cpp	Added RST block example with proper markers
docs/source/components/rst_parser.rst	New documentation describing the simplified RST parser and its limitations
docs/source/components/configuration.rst	Documented new configuration options for RST parsing
docs/source/components/analyse.rst	Updated documentation with RST block examples
docs/source/components/directive.rst	Updated to reflect support for both one-line and RST block needs
docs/source/development/change_log.rst	Added release notes for new RST parsing feature
pyproject.toml	Added Lark dependency

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ubmarco]

After running the tests I find a new file in my worktree invalid_objs.json in the repo root.

[ubmarco]

This valid and basic RST

.. spec:: comment in content
    :option: value
    :option2: value2

    Content without blank line.

leads to UnexpectedToken() parsing result.

[juiwenchen]

This valid and basic RST

.. spec:: comment in content
    :option: value
    :option2: value2

    Content without blank line.

leads to UnexpectedToken() parsing result.

Indentation spaces need to be configured if it's not 3 by default

https://github.com/useblocks/sphinx-codelinks/pull/44/files#diff-2f95c5da4f0367f21172743cf633c7513b5b1898a7c79bfdb690d37bcae882d8R498

[MrChocolateMoose]

I have a bug report from this feature.

I just started using sphinx-test-reports and wanted to convert my custom user defined test case type to test-case.

When I use that as part of marked-rst, like so:

/*
@rst

.. test-case:: test_xyz
   :file: test_reports/abc.xml
   :suite: test_abc
   :case: test_xyz
   :id: SW_TEST_CASE_XYZ
   :release: abc-1.2.3
   :uplink: SW_UNIT_IF_XYZ

   Test case to verify xyz does behavior abc

@endrst
*/
void test_xyz(void)
{
   ...
}

It is not working properly and it emits:

InvalidNeedException: Unknown need type 'test-case'. [invalid_type]

If I use it in a rst file, then it works fine so I know it's not the syntax.
Other needs whose types are defined in toml file are working fine as I indicated above.

[juiwenchen]

I have a bug report from this feature.

I just started using sphinx-test-reports and wanted to convert my custom user defined test case type to test-case.

When I use that as part of marked-rst, like so:

/*
@rst

.. test-case:: test_xyz
   :file: test_reports/abc.xml
   :suite: test_abc
   :case: test_xyz
   :id: SW_TEST_CASE_XYZ
   :release: abc-1.2.3
   :uplink: SW_UNIT_IF_XYZ

   Test case to verify xyz does behavior abc

@endrst
*/
void test_xyz(void)
{
   ...
}

It is not working properly and it emits:

InvalidNeedException: Unknown need type 'test-case'. [invalid_type]

If I use it in a rst file, then it works fine so I know it's not the syntax. Other needs whose types are defined in toml file are working fine as I indicated above.

It is super interesting.

The directive parser extracts the directive test-case correctly and feed it to add_need(). However, test-case is only a Sphinx directive instead of a need type. It works in RST because, Sphinx calls test-case directive, and it generates needs with type testcase according to the given options in this directive.

The updated documentation of this PR mentioned the limitation of this parser, but it could be misleading for users to think it can do things like what Sphinx does. I will talk to my colleagues about it, and thank you so much for your feedback.