NAME

perl5240cdelta - what is new for cperl v5.24.0

DESCRIPTION

This document describes perl-only differences between the cperl 5.22.3 release and the cperl 5.24.0 release.

Core Enhancements

Core Enhancements from Perl5

Backported some fixes from p5p, merged with perl-5.24.0. [cperl #137]

Security

DynaLoader format string hardening

Replace all % characters in user-controlled library filenames, passed via the system dl_error call verbatim to printf, without any arguments on the stack, which could lead to execution of arbitrary stack code. No CVE. This affects all systems with dynamic loading where the attacker can cause a dynamic loading error.

CVSSv2 Severity: 7.2 (AV:L/AC:L/Au:N/C:C/I:C/A:C/E:U/RL:OF/RC:C/CDP:MH/TD:H/CR:H/IR:H/AR:ND)

No Unicode confusables +UFFA0, +U3164

In deviation from Unicode 1.1 we treat the two HANGUL FILLER characters +UFFA0 and +U3164 not as valid ID_Start and ID_Continue characters for perl identifiers. Variable and package names may not begin with them and may not contain them.

They are usually rendered as whitespace, and would lead to classic TR39 confusables. See https://github.com/jagracey/Awesome-Unicode#user-content-variable-identifiers-can-effectively-include-whitespace and http://www.unicode.org/reports/tr39/.

In a more Korean friendly environment, we could check for a ID_Start Hangul filler if the next character is a valid Hangul ID_Continue character, and allow it then. Ditto for a ID_Continue Hangul filler if the previous and next character is a valid Hangul ID_Start or ID_Continue character, and allow it then. But those fillers should be treated as whitespace, and should be ignored.

http://www.unicode.org/L2/L2006/06310-hangul-decompose9.pdf explains:

The two other hangul fillers HANGUL CHOSEONG FILLER (Lf), i.e. lead filler, and HANGUL JUNGSEONG FILLER (Vf) are used as placeholders for missing letters, where there should be at least one letter.

... that leaves the (HALFWIDTH) HANGUL FILLERs useless. Indeed, they should not be rendered at all, despite that they have been given the property Lo. Note that these FILLERs are also given the property of Default_Ignorable_Codepoint.

Note that the standard normal forms NFKD and NFKC ... return (in all views) incorrect results for strings containing these characters.

XSLoader relative paths with eval or #line

Upstream XSLoader 0.22 (perl 5.26) fixed a minor security problem with XSLoader within eval or with a #line directive, which can load a local relative shared library, which is not in @INC. See [cpan #115808].

cperl XSLoader was already protected against the eval case since 5.22, when being rewritten in C. cperl-5.24.0 fixed now also ignoring a relative filename in a #line directive, when the relative path is not in @INC.

Modules and Pragmata

New Modules and Pragmata

fake_signatures 0.01

Prohibit the automatic conversion to fake signatures via no fake_signatures, and document the new conversion of argument handling to fake signatures. See fake_signatures.

Updated Modules and Pragmata

Many internal core modules are now half-way "modernized", i.e. use typed signatures, and catch errors at compile-time. [cperl #97]

But no classes, methods and roles yet.

XSLoader 1.02c

Fixed the #line relativefilename part of the [cpan #115808] security problem, the (eval 1) part was already fixed in the cperl rewrite as XS.

Ensure that the local stash filename part is absolute and ends with /, the part before "auto/" and before the fallback to the DynaLoader search.

Net::Cmd

Fixed utf8 handling, suse [bnc#493978]

Note that libnet has hundreds of more open tickets

Pod::Perldoc

Favor nroff over groff, suse perl-nroff.diff [bnc#463444]

Socket 2.021_02

Silence some CC warnings

Fix my to our $XS_VERSION

Check for max length before derefing by length (jhi) [cpan #111707]

DynaLoader 2.04c

See "DynaLoader format string hardening" above.

B::C 1.54_08

Many Windows and cygwin fixes, for the old MSVC compiler, and PERL_CORE test integration. Use many new core exports needed for windows.

Update perlcc to 2.21, handle multiple -I and -L arguments.

Handle cperl OP_SIGNATURE.

Fix refcount of cop hints hashes with 5.22-nt [cperl #220]

Better UVX L and U suffices.

Handle shared IV and UV xpv structs, fixed 5.24 specific assertions, with shared xpviv/xpvuv. fixed wrong 32 bit ptr offset. re-enabled all 32bit tests.

Handle changed subdirs-test target with newer EUMM.

B::Deparse 1.37_02c

Add more cperl-specific ops: aelem*_u, u_{add,subtract,multiply}, {i,n,s}_aelem{,_u}

Test::Simple 1.401014c

Based on 1.001014, not Test2 as in 1.302022 yet. Test2 breaks too many modules still, and I need to time to properly modernize it (as v2.x)

Modernized and stricter type checks. skip does the switched argument check at compile-time. See the FAQ at http://perl11.org/cperl/STATUS.html why we enforce stricter types there.

Enable t/is_deeply_with_threads.t

File-Path 3.12_01c

Modernized. Enforce many str types.

PathTools 4.63c

Modernized. Enforce many str types.

Enable XS prototypes and suppress warnings. [cperl #152]

Disallow the optional getcwd argument (5.8.5 problem). Cannot be called as method.

Pod-Simple 4.30c

Modernized

Pod-Html 2.22c

Modernized

bignum 0.42c

Modernized

bigint 0.43c

Modernized. Fixed deprecated usage of ::binf, ::bnan as function.

bigrat 0.42c

Modernized

CPAN

Fixed tests. Type skip calls and File-Path args.

Change FirstTime yaml_module default from YAML to YAML::XS and prohibit the unsupported CPAN::Meta::YAML.

ExtUtils-MakeMaker

Type File-Path args.

Fix ExtUtils::Liblist::Kid for systemlib symlinks, esp. on darwin.

Parse-CPAN-Meta 1.5000c

Modernized. str type for skip

CPAN-Meta 2.150005c

fix Encode test for modernized Test-Simple, stringify the 1st skip arg

CPAN::Meta::Requirements 3.140c
Test-Harness 3.36

type the skip calls

Hash-Util-FieldHash

type the skip calls

YAML-LibYAML 0.70

Spiffy and Test::Base 0.88c fixes for fake signatures.

Spiffy uses a source filter to expand many methods, but this does not work with fake_signatures. Expand the compile-time state of Test::Base to avoid source filters. Also add . to @INC in some tests.

utf-8 handling stills throws many warnings.

Filter-Util-Call 0.92

Fixed 1 test for signature errmsg

IO-Compress

Fixed 1 test for signature errmsg

Devel-Peek 1.23

Skip tests for changed dynamic prototype() with sigs. Fix tests with -DDEBUG_LEAKING_SCALARS

B 1.62_02

add B::OP::rettype, B::OP::typechecked, B::CV::SIGOP and support for OP_SIGNATURE with B::UNOP_AUX::aux_list and B::UNOP_AUX::string.

Note: With threads in B::UNOP_AUX::aux_list the padoffsets are returned as such, and not as SV. This is different to upstream perl5, a perl5 bug.

Devel-NYTProf 6.03

fix test for short CV_NAME_NOMAIN. silence a C compiler warning.

perl5db 1.49_04c

fix test for short CV_NAME_NOMAIN. run-time load IO::Handle, broke miniperl. handle calls to signatures via goto, and add tests.

Digest::MD5 2.55_01
warnings 1.35

Add category types.

Disable 3 experimental warnings which are now default with cperl.

Archive::Tar 2.08

Added t/09_roundtrip.t

Config 6.21

Various minor fixes

Config::Perl::V 0.26_01

cperl patches. Add tests for 5.22 and 5.24. Support some multi-line keys: ccflags ldflags lddlflags Changed tests to use done_testing().

Cpanel::JSON::XS 3.0217

- Improve test 20_unknown for older JSON::PP (madsen)

- Fix wrong test 117 (pghmcfc)

- TODO the fragile mojo boolean interop test.

- Improve error message with class based method calls, when forgetting ->new. [cperl #66]

- Fix a off-by-one IV_MIN -> NV overflow in decode_json. [cperl #67] (xdg)

- Avoid encode_sv SEGV with -Dusequadmath [cperl #62] Fix quadmath NV stringification.

DB_File 1.838

Skip checking the return status of the api calls.

Devel::PPPort 3.33_02

Support the latest versions.

cperl specific: Minor -Dfortify_inc fixes, kept our Hv macros, fix __attribute__((warn_unused_result)) on windows.

HTTP::Tiny 0.058

Added some methods.

IPC::SysV 2.07

Sync with upstream, they fixed [RT #112827]

JSON::PP 2.27400

Some fixes. Kept our cperl-specific fallbacks to Cpanel::JSON::XS and boolean interop.

Locale::Codes 3.39

Removed the deprecated alias_code subroutine. Added some LOCALE_CODE_* UN and GENC constants.

Math::BigInt 1.999722

Many changes. Fixed a skip count/why mixup in t/mbimbf.inc, detected by cperl.

Math::BigInt::FastCalc 0.42

No changes.

Math::BigRat 0.260804

Many changes.

Module-Metadata 1.000031-TRIAL

Fixed the t/extract-version.t TODO tests.

libnet 3.08

Keeping our darwin performance fix for hostname

Pod::Checker 1.73

Now based on Pod::Simple, not on Pod::Parser anymore. dos2unix

Pod::Functions 1.11

Whitespace only to please the new Pod::Checker

Pod::Usage 1.69_01

Remove the Pod::Find test, using pod_where. dos2unix

Sys::Syslog 0.34

Handle fallback/syslog.h properly. honor HAVE_SETLOCALE fix windows fallback. Exclude the t/facilities-routing.t test requiring POE. Skip failing udp tests when the server is overloaded.

Time::HiRes 1.9733

Minor changes from 5.22.3c. Improve t/nanosleep.t to show smoker errors.

Thread::Queue 3.11

Replace threads::yield with thread->join in t/07_lock.t.

threads 2.09

Updated from 2.06. Improve 2 tests

threads::shared 1.52

No functional changes

Term::ReadKey

Improve Makefile.PL, esp. for Windows. See https://github.com/jonathanstowe/TermReadKey/pull/15

Scalar-List-Utils 1.45_04

Improved set_subname for ' package seperators. ' is counted as seperator only if it appears at first, before any ::. If seen, do the same magic as parse_gv_stash_name, converting all seperators over to ::. Taken with improvements from Sub::Name.

Data::Dumper 2.162

Fix high isASCII chars for EBCDIC, [cpan #12282], on CPAN as 2.161.

Fix [perl #128524], wrong indentation for utf-8 key hash elements.

Not Updated Modules and Pragmata

This modules are currently not ready to be updated.

MakeUtils::MakeMaker 7.11_06

Broken for -Dfortify_inc, no "." in @INC. Broken new "_" logic when merged with our 'c' suffix logic.

Test2

We need some time to modernize it, as done with the previous version.

Our Test::Simple and Test::More versions were bumped to 1.4 to prohibit automatic installation of Test2 from CPAN, to use our improved versions instead.

version 0.9917

Broken new "_" logic when merged with our 'c' suffix logic.

Diagnostics

The following additions or changes have been made to diagnostic output, including warnings and fatal error messages. For the complete list of diagnostic messages, see perldiag.

New Diagnostics

New Errors

Not enough arguments for %s

(F) The function requires more arguments than you specified.

This error might be detected at compile-time or run-time.

If the called function is declared with a signature, and the error message includes "Want: \d, but got: 0", it might be that the call &$sub without () needs to be replaced with a goto \⊂. See "Changed calls to signatures" in perlcperl.

Invalid type %s for %s %s

(F) The function cannot handle the declared type of the argument. E.g. tie or bless cannot use a typed arg with a coretype, as a coretype declaration guarantees no magic.

Reference parameter cannot take default value

(F) In a subroutine signature, you put a = after a reference \ parameter. A reference parameter cannot be assigned default values on subroutine entries.

Subroutine signature has more than %d parameters

(F) A subroutine signature had more than 32767 individual parameter entries.. In the unlikely event that you need so many parameters, use a slurpy array instead.

New Warnings

Wrong type %s, expected %s

(W types) The type of the argument does not match the user-defined type of the function declaration.

Inserting type cast %s to %s

(W types) The compiler found an argument with a declared type, which the needs to be converted at run-time. This warning usually points to a wrong type usage.

Useless use of :caller, did you mean :-caller instead?

(W misc) The ":caller" attribute has no effect in pure perl subroutine declarations. You applied it to a subroutine via attributes.pm. This is only valid as negated variant :-caller on subroutines.

Platform Support

Platform-Specific Notes

Win32

Work is ongoing for perlcc compiler support on Win32. It is now usable, but not as good as on other platforms.

Darwin

With -DDEBUGGING dsymutil is called automatically on all created binaries for easier debugging.

Cygwin

Several fixes were added: silence time64.h compilation warnings, fixup archname for 64bit, set d_eofnblk correctly for non-win95, define _GNU_SOURCE to declare memmem.

See RT #128358, RT #128359 and RT #127708.

Internal Changes

Selected Bug Fixes

handle method calls on protected stashes

cperl #171

Known bug upstream, not fixed there. This problem appears more often with cperl with its protected coretypes than upstream.

fedora: Do not crash when inserting a non-stash into a stash

RT#128238

Fedora Patch 37

fedora: Do not treat %: as a stash

RT#128238

Fedora Patch36

fedora: Fix precedence in hv_ename_delete

RT#128086

Fedora Patch35

fedora: Do not use unitialized memory in $h{\const} warnings

RT#128189

Fedora Patch34

fedora: Do not mangle errno from failed socket calls

RT#128316

Fedora Patch32

fedora: Backport memory leak when compiling a regular expression with a POSIX class

E.g. when use re 'strict'; is used.

RT#128313

Fedora Patch31

suse: perl5.24.0.dif

Many Configure and linux hints enhancements, esp for lib64, probe fixes, gdbm and ODBM fixes, gnu readline integration with the debugger. See https://build.opensuse.org/package/show/devel:languages:perl/perl

suse: fix regexp backref overflows

With many backref groups (>I32)

suse: perl-saverecontext.diff RT#76538

Handle get magic with globs in the regex compiler. Correctly restore context, esp. when loading unicode swashes. Reported at 5.12, patched for suse 5.14, still ignored with 5.24.

Issues with (?0) being wrong in 5.24.0, reginsert recursion

The pattern /\X*(?0)/ and various flavours thereof did not work correctly due to RExC_open_paren[0] being inappropriately adjusted. Broken in cperl-5.24.0-RC1-2 and perl-5.24.0 upstream.

See RT #128109 and RT #128085.

locale race condition when switching to C for error messages

A race condition which occurred when computing "$!" with threads activated has been fixed. This showed up only on Darwin platforms. A related problem on Cygwin platforms involving UTF-8 strings has also been fixed. See RT #127708

Fix crash with @a &.= etc.

See RT #128204

Fix leak in c?perl 5.24.0 with strict and regex posix char classes

See RT #128313

Fix PERLIO=stdio dup on empty filehandle

See RT #63244 or http://bugs.debian.org/164615, crashes from 5.8.8 to blead.

env PERLIO=stdio perl -e'open(F, "<&STDOUT")' 1</dev/null

Fix crash from empty curcop in gp_free/ckWARN_d

See RT #128597.

open my $fh, ">", \$buf; my $sub = eval q|sub {die}|; $sub->()' crashes from 5.22 to 5.24.1.

Fix crash when vivifying stub in deleted pkg

See RT #128532.

delete $My::{"Foo::"}; \&My::Foo::foo crashes from 5.18 to 5.24.1.

Acknowledgements

cperl 5.24.0 represents approximately 14 weeks of development since cperl 5.22.2c and contains approximately 1,100,000 lines of changes across 1,600 files from 59 authors.

Excluding auto-generated files, documentation and release tools, there were approximately 130,000 lines of changes to 700 .pm, .t, .c and .h files.

The following people are known to have contributed the improvements that became cperl 5.24.0:

Karl Williamson, Reini Urban, David Mitchell, Jarkko Hietaniemi, Andy Broad, Tony Cook, Daniel Dragan, Lukas Mai, Craig A. Berry, Yves Orton, Aaron Crane, Father Chrysostomos, Ricardo Signes, Dagfinn Ilmari Mannsåker, Steve Hay, H.Merijn Brand, James E Keenan, Matthew Horsfall, Aristotle Pagaltzis, Karen Etheridge, Abigail, Chris 'BinGOs' Williams, Shlomi Fish, Doug Bell, Dan Collins, Stevan Little, Tom Hukins, Rafael Garcia-Suarez, Andy Dougherty, Ed J, Vincent Pit, Todd Rinaldo, Nicholas Clark, Leon Timmermans, Thomas Sibley, Achim Gratz, Vladimir Timofeev, Stanislaw Pusep, Aaron Priven, Dr.Ruud, Pip Cet, Hugo van der Sanden, Mohammed El-Afifi, Mattia Barbon, John SJ Anderson, Victor Adam, Alex Vandiver, John Peacock, Niko Tyni, Alexander D'Archangel, Ludovic E. R. Tolhurst-Cleaver, Zachary Storer, Herbert Breunung, Ed Avis, Sisyphus, Max Maischein, Ivan Pozdeev, Chas. Owens, Peter Martini.

The list above is almost certainly incomplete as it is automatically generated from version control history including the perl and cperl repos. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the Perl bug tracker and the cperl github issues.

Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.

For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.

Reporting Bugs

If you find what you think is a bug, you might check the articles recently posted to the comp.lang.perl.misc newsgroup and the perl bug database at https://rt.perl.org/ . There may also be information at http://www.perl.org/ , the Perl Home Page.

If you believe you have an unreported bug, please run the perlbug program included with your release. Be sure to trim your bug down to a tiny but sufficient test case. Your bug report, along with the output of perl -V, will be sent off to perlbug@perl.org to be analysed by the Perl porting team.

If you think it's a cperl specific bug or trust the cperl developers more please file an issue at https://github.com/perl11/cperl/issues.

If the bug you are reporting has security implications, which make it inappropriate to send to a publicly archived mailing list, then please send it to perl5-security-report@perl.org. This points to a closed subscription unarchived mailing list, which includes all the p5p core committers, who will be able to help assess the impact of issues, figure out a resolution, and help co-ordinate the release of patches to mitigate or fix the problem across all platforms on which Perl is supported. Please only use this address for security issues in the Perl core, not for modules independently distributed on CPAN.

If you trust the cperl developers more, please send an email to them. The p5p security teams skips many security issues, or are unwilling to fix them.

SEE ALSO

The Changes file for an explanation of how to view exhaustive details on what changed.

The INSTALL file for how to build Perl.

The README file for general stuff.

The Artistic and Copying files for copyright information.